When Benchmarks are Targets: Revealing the Sensitivity of Large Language Model Leaderboards
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866929407126405120 |
|---|---|
| author | Alzahrani, Norah Alyahya, Hisham Abdullah Alnumay, Yazeed Alrashed, Sultan Alsubaie, Shaykhah Almushaykeh, Yusef Mirza, Faisal Alotaibi, Nouf Altwairesh, Nora Alowisheq, Areeb Bari, M Saiful Khan, Haidar |
| author_facet | Alzahrani, Norah Alyahya, Hisham Abdullah Alnumay, Yazeed Alrashed, Sultan Alsubaie, Shaykhah Almushaykeh, Yusef Mirza, Faisal Alotaibi, Nouf Altwairesh, Nora Alowisheq, Areeb Bari, M Saiful Khan, Haidar |
| contents | Large Language Model (LLM) leaderboards based on benchmark rankings are regularly used to guide practitioners in model selection. Often, the published leaderboard rankings are taken at face value - we show this is a (potentially costly) mistake. Under existing leaderboards, the relative performance of LLMs is highly sensitive to (often minute) details. We show that for popular multiple-choice question benchmarks (e.g., MMLU), minor perturbations to the benchmark, such as changing the order of choices or the method of answer selection, result in changes in rankings up to 8 positions. We explain this phenomenon by conducting systematic experiments over three broad categories of benchmark perturbations and identifying the sources of this behavior. Our analysis results in several best-practice recommendations, including the advantage of a hybrid scoring method for answer selection. Our study highlights the dangers of relying on simple benchmark evaluations and charts the path for more robust evaluation schemes on the existing benchmarks. The code for this paper is available at https://github.com/National-Center-for-AI-Saudi-Arabia/lm-evaluation-harness. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_01781 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | When Benchmarks are Targets: Revealing the Sensitivity of Large Language Model Leaderboards Alzahrani, Norah Alyahya, Hisham Abdullah Alnumay, Yazeed Alrashed, Sultan Alsubaie, Shaykhah Almushaykeh, Yusef Mirza, Faisal Alotaibi, Nouf Altwairesh, Nora Alowisheq, Areeb Bari, M Saiful Khan, Haidar Computation and Language Artificial Intelligence Machine Learning Large Language Model (LLM) leaderboards based on benchmark rankings are regularly used to guide practitioners in model selection. Often, the published leaderboard rankings are taken at face value - we show this is a (potentially costly) mistake. Under existing leaderboards, the relative performance of LLMs is highly sensitive to (often minute) details. We show that for popular multiple-choice question benchmarks (e.g., MMLU), minor perturbations to the benchmark, such as changing the order of choices or the method of answer selection, result in changes in rankings up to 8 positions. We explain this phenomenon by conducting systematic experiments over three broad categories of benchmark perturbations and identifying the sources of this behavior. Our analysis results in several best-practice recommendations, including the advantage of a hybrid scoring method for answer selection. Our study highlights the dangers of relying on simple benchmark evaluations and charts the path for more robust evaluation schemes on the existing benchmarks. The code for this paper is available at https://github.com/National-Center-for-AI-Saudi-Arabia/lm-evaluation-harness. |
| title | When Benchmarks are Targets: Revealing the Sensitivity of Large Language Model Leaderboards |
| topic | Computation and Language Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2402.01781 |