Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2504.18838 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910920233451520 |
|---|---|
| author | Cao, Yixin Hong, Shibo Li, Xinze Ying, Jiahao Ma, Yubo Liang, Haiyuan Liu, Yantao Yao, Zijun Wang, Xiaozhi Huang, Dan Zhang, Wenxuan Huang, Lifu Chen, Muhao Hou, Lei Sun, Qianru Ma, Xingjun Wu, Zuxuan Kan, Min-Yen Lo, David Zhang, Qi Ji, Heng Jiang, Jing Li, Juanzi Sun, Aixin Huang, Xuanjing Chua, Tat-Seng Jiang, Yu-Gang |
| author_facet | Cao, Yixin Hong, Shibo Li, Xinze Ying, Jiahao Ma, Yubo Liang, Haiyuan Liu, Yantao Yao, Zijun Wang, Xiaozhi Huang, Dan Zhang, Wenxuan Huang, Lifu Chen, Muhao Hou, Lei Sun, Qianru Ma, Xingjun Wu, Zuxuan Kan, Min-Yen Lo, David Zhang, Qi Ji, Heng Jiang, Jing Li, Juanzi Sun, Aixin Huang, Xuanjing Chua, Tat-Seng Jiang, Yu-Gang |
| contents | Large Language Models (LLMs) are advancing at an amazing speed and have become indispensable across academia, industry, and daily applications. To keep pace with the status quo, this survey probes the core challenges that the rise of LLMs poses for evaluation. We identify and analyze two pivotal transitions: (i) from task-specific to capability-based evaluation, which reorganizes benchmarks around core competencies such as knowledge, reasoning, instruction following, multi-modal understanding, and safety; and (ii) from manual to automated evaluation, encompassing dynamic dataset curation and "LLM-as-a-judge" scoring.
Yet, even with these transitions, a crucial obstacle persists: the evaluation generalization issue. Bounded test sets cannot scale alongside models whose abilities grow seemingly without limit. We will dissect this issue, along with the core challenges of the above two transitions, from the perspectives of methods, datasets, evaluators, and metrics. Due to the fast evolving of this field, we will maintain a living GitHub repository (links are in each section) to crowd-source updates and corrections, and warmly invite contributors and collaborators. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_18838 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarks Cao, Yixin Hong, Shibo Li, Xinze Ying, Jiahao Ma, Yubo Liang, Haiyuan Liu, Yantao Yao, Zijun Wang, Xiaozhi Huang, Dan Zhang, Wenxuan Huang, Lifu Chen, Muhao Hou, Lei Sun, Qianru Ma, Xingjun Wu, Zuxuan Kan, Min-Yen Lo, David Zhang, Qi Ji, Heng Jiang, Jing Li, Juanzi Sun, Aixin Huang, Xuanjing Chua, Tat-Seng Jiang, Yu-Gang Computation and Language Large Language Models (LLMs) are advancing at an amazing speed and have become indispensable across academia, industry, and daily applications. To keep pace with the status quo, this survey probes the core challenges that the rise of LLMs poses for evaluation. We identify and analyze two pivotal transitions: (i) from task-specific to capability-based evaluation, which reorganizes benchmarks around core competencies such as knowledge, reasoning, instruction following, multi-modal understanding, and safety; and (ii) from manual to automated evaluation, encompassing dynamic dataset curation and "LLM-as-a-judge" scoring. Yet, even with these transitions, a crucial obstacle persists: the evaluation generalization issue. Bounded test sets cannot scale alongside models whose abilities grow seemingly without limit. We will dissect this issue, along with the core challenges of the above two transitions, from the perspectives of methods, datasets, evaluators, and metrics. Due to the fast evolving of this field, we will maintain a living GitHub repository (links are in each section) to crowd-source updates and corrections, and warmly invite contributors and collaborators. |
| title | Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarks |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2504.18838 |