Easy2Hard-Bench: Standardized Difficulty Labels for Profiling LLM Performance and Generalization
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866912420168990720 |
|---|---|
| author | Ding, Mucong Deng, Chenghao Choo, Jocelyn Wu, Zichu Agrawal, Aakriti Schwarzschild, Avi Zhou, Tianyi Goldstein, Tom Langford, John Anandkumar, Anima Huang, Furong |
| author_facet | Ding, Mucong Deng, Chenghao Choo, Jocelyn Wu, Zichu Agrawal, Aakriti Schwarzschild, Avi Zhou, Tianyi Goldstein, Tom Langford, John Anandkumar, Anima Huang, Furong |
| contents | While generalization over tasks from easy to hard is crucial to profile language models (LLMs), the datasets with fine-grained difficulty annotations for each problem across a broad range of complexity are still blank. Aiming to address this limitation, we present Easy2Hard-Bench, a consistently formatted collection of 6 benchmark datasets spanning various domains, such as mathematics and programming problems, chess puzzles, and reasoning questions. Each problem within these datasets is annotated with numerical difficulty scores. To systematically estimate problem difficulties, we collect abundant performance data on attempts to each problem by humans in the real world or LLMs on the prominent leaderboard. Leveraging the rich performance data, we apply well-established difficulty ranking systems, such as Item Response Theory (IRT) and Glicko-2 models, to uniformly assign numerical difficulty scores to problems. Moreover, datasets in Easy2Hard-Bench distinguish themselves from previous collections by a higher proportion of challenging problems. Through extensive experiments with six state-of-the-art LLMs, we provide a comprehensive analysis of their performance and generalization capabilities across varying levels of difficulty, with the aim of inspiring future research in LLM generalization. The datasets are available at https://huggingface.co/datasets/furonghuang-lab/Easy2Hard-Bench. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_18433 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Easy2Hard-Bench: Standardized Difficulty Labels for Profiling LLM Performance and Generalization Ding, Mucong Deng, Chenghao Choo, Jocelyn Wu, Zichu Agrawal, Aakriti Schwarzschild, Avi Zhou, Tianyi Goldstein, Tom Langford, John Anandkumar, Anima Huang, Furong Machine Learning Artificial Intelligence Computation and Language While generalization over tasks from easy to hard is crucial to profile language models (LLMs), the datasets with fine-grained difficulty annotations for each problem across a broad range of complexity are still blank. Aiming to address this limitation, we present Easy2Hard-Bench, a consistently formatted collection of 6 benchmark datasets spanning various domains, such as mathematics and programming problems, chess puzzles, and reasoning questions. Each problem within these datasets is annotated with numerical difficulty scores. To systematically estimate problem difficulties, we collect abundant performance data on attempts to each problem by humans in the real world or LLMs on the prominent leaderboard. Leveraging the rich performance data, we apply well-established difficulty ranking systems, such as Item Response Theory (IRT) and Glicko-2 models, to uniformly assign numerical difficulty scores to problems. Moreover, datasets in Easy2Hard-Bench distinguish themselves from previous collections by a higher proportion of challenging problems. Through extensive experiments with six state-of-the-art LLMs, we provide a comprehensive analysis of their performance and generalization capabilities across varying levels of difficulty, with the aim of inspiring future research in LLM generalization. The datasets are available at https://huggingface.co/datasets/furonghuang-lab/Easy2Hard-Bench. |
| title | Easy2Hard-Bench: Standardized Difficulty Labels for Profiling LLM Performance and Generalization |
| topic | Machine Learning Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2409.18433 |