Changing Answer Order Can Decrease MMLU Accuracy
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866912112740139008 |
|---|---|
| author | Gupta, Vipul Pantoja, David Ross, Candace Williams, Adina Ung, Megan |
| author_facet | Gupta, Vipul Pantoja, David Ross, Candace Williams, Adina Ung, Megan |
| contents | As large language models (LLMs) have grown in prevalence, particular benchmarks have become essential for the evaluation of these models and for understanding model capabilities. Most commonly, we use test accuracy averaged across multiple subtasks in order to rank models on leaderboards, to determine which model is best for our purposes. In this paper, we investigate the robustness of the accuracy measurement on a widely used multiple choice question answering dataset, MMLU. When shuffling the answer label contents, we find that all explored models decrease in accuracy on MMLU, but not every model is equally sensitive. These findings suggest a possible adjustment to the standard practice of leaderboard testing, where we additionally consider the percentage of examples each model answers correctly by random chance. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_19470 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Changing Answer Order Can Decrease MMLU Accuracy Gupta, Vipul Pantoja, David Ross, Candace Williams, Adina Ung, Megan Computation and Language As large language models (LLMs) have grown in prevalence, particular benchmarks have become essential for the evaluation of these models and for understanding model capabilities. Most commonly, we use test accuracy averaged across multiple subtasks in order to rank models on leaderboards, to determine which model is best for our purposes. In this paper, we investigate the robustness of the accuracy measurement on a widely used multiple choice question answering dataset, MMLU. When shuffling the answer label contents, we find that all explored models decrease in accuracy on MMLU, but not every model is equally sensitive. These findings suggest a possible adjustment to the standard practice of leaderboard testing, where we additionally consider the percentage of examples each model answers correctly by random chance. |
| title | Changing Answer Order Can Decrease MMLU Accuracy |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2406.19470 |