Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | https://arxiv.org/abs/2406.04127 |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866916560348643328 |
|---|---|
| author | Gema, Aryo Pradipta Leang, Joshua Ong Jun Hong, Giwon Devoto, Alessio Mancino, Alberto Carlo Maria Saxena, Rohit He, Xuanli Zhao, Yu Du, Xiaotang Madani, Mohammad Reza Ghasemi Barale, Claire McHardy, Robert Harris, Joshua Kaddour, Jean van Krieken, Emile Minervini, Pasquale |
| author_facet | Gema, Aryo Pradipta Leang, Joshua Ong Jun Hong, Giwon Devoto, Alessio Mancino, Alberto Carlo Maria Saxena, Rohit He, Xuanli Zhao, Yu Du, Xiaotang Madani, Mohammad Reza Ghasemi Barale, Claire McHardy, Robert Harris, Joshua Kaddour, Jean van Krieken, Emile Minervini, Pasquale |
| contents | Maybe not. We identify and analyse errors in the popular Massive Multitask Language Understanding (MMLU) benchmark. Even though MMLU is widely adopted, our analysis demonstrates numerous ground truth errors that obscure the true capabilities of LLMs. For example, we find that 57% of the analysed questions in the Virology subset contain errors. To address this issue, we introduce a comprehensive framework for identifying dataset errors using a novel error annotation protocol. Then, we create MMLU-Redux, which is a subset of 5,700 manually re-annotated questions across all 57 MMLU subjects. We estimate that 6.49% of MMLU questions contain errors. Using MMLU-Redux, we demonstrate significant discrepancies with the model performance metrics that were originally reported. Our results strongly advocate for revising MMLU's error-ridden questions to enhance its future utility and reliability as a benchmark. https://huggingface.co/datasets/edinburgh-dawg/mmlu-redux-2.0. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_04127 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Are We Done with MMLU? Gema, Aryo Pradipta Leang, Joshua Ong Jun Hong, Giwon Devoto, Alessio Mancino, Alberto Carlo Maria Saxena, Rohit He, Xuanli Zhao, Yu Du, Xiaotang Madani, Mohammad Reza Ghasemi Barale, Claire McHardy, Robert Harris, Joshua Kaddour, Jean van Krieken, Emile Minervini, Pasquale Computation and Language Artificial Intelligence Maybe not. We identify and analyse errors in the popular Massive Multitask Language Understanding (MMLU) benchmark. Even though MMLU is widely adopted, our analysis demonstrates numerous ground truth errors that obscure the true capabilities of LLMs. For example, we find that 57% of the analysed questions in the Virology subset contain errors. To address this issue, we introduce a comprehensive framework for identifying dataset errors using a novel error annotation protocol. Then, we create MMLU-Redux, which is a subset of 5,700 manually re-annotated questions across all 57 MMLU subjects. We estimate that 6.49% of MMLU questions contain errors. Using MMLU-Redux, we demonstrate significant discrepancies with the model performance metrics that were originally reported. Our results strongly advocate for revising MMLU's error-ridden questions to enhance its future utility and reliability as a benchmark. https://huggingface.co/datasets/edinburgh-dawg/mmlu-redux-2.0. |
| title | Are We Done with MMLU? |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2406.04127 |