Saved in:
| Main Authors: | , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2508.15361 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916911132966912 |
|---|---|
| author | Ni, Shiwen Chen, Guhong Li, Shuaimin Chen, Xuanang Li, Siyi Wang, Bingli Wang, Qiyao Wang, Xingjian Zhang, Yifan Fan, Liyang Li, Chengming Xu, Ruifeng Sun, Le Yang, Min |
| author_facet | Ni, Shiwen Chen, Guhong Li, Shuaimin Chen, Xuanang Li, Siyi Wang, Bingli Wang, Qiyao Wang, Xingjian Zhang, Yifan Fan, Liyang Li, Chengming Xu, Ruifeng Sun, Le Yang, Min |
| contents | In recent years, with the rapid development of the depth and breadth of large language models' capabilities, various corresponding evaluation benchmarks have been emerging in increasing numbers. As a quantitative assessment tool for model performance, benchmarks are not only a core means to measure model capabilities but also a key element in guiding the direction of model development and promoting technological innovation. We systematically review the current status and development of large language model benchmarks for the first time, categorizing 283 representative benchmarks into three categories: general capabilities, domain-specific, and target-specific. General capability benchmarks cover aspects such as core linguistics, knowledge, and reasoning; domain-specific benchmarks focus on fields like natural sciences, humanities and social sciences, and engineering technology; target-specific benchmarks pay attention to risks, reliability, agents, etc. We point out that current benchmarks have problems such as inflated scores caused by data contamination, unfair evaluation due to cultural and linguistic biases, and lack of evaluation on process credibility and dynamic environments, and provide a referable design paradigm for future benchmark innovation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_15361 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | A Survey on Large Language Model Benchmarks Ni, Shiwen Chen, Guhong Li, Shuaimin Chen, Xuanang Li, Siyi Wang, Bingli Wang, Qiyao Wang, Xingjian Zhang, Yifan Fan, Liyang Li, Chengming Xu, Ruifeng Sun, Le Yang, Min Computation and Language In recent years, with the rapid development of the depth and breadth of large language models' capabilities, various corresponding evaluation benchmarks have been emerging in increasing numbers. As a quantitative assessment tool for model performance, benchmarks are not only a core means to measure model capabilities but also a key element in guiding the direction of model development and promoting technological innovation. We systematically review the current status and development of large language model benchmarks for the first time, categorizing 283 representative benchmarks into three categories: general capabilities, domain-specific, and target-specific. General capability benchmarks cover aspects such as core linguistics, knowledge, and reasoning; domain-specific benchmarks focus on fields like natural sciences, humanities and social sciences, and engineering technology; target-specific benchmarks pay attention to risks, reliability, agents, etc. We point out that current benchmarks have problems such as inflated scores caused by data contamination, unfair evaluation due to cultural and linguistic biases, and lack of evaluation on process credibility and dynamic environments, and provide a referable design paradigm for future benchmark innovation. |
| title | A Survey on Large Language Model Benchmarks |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2508.15361 |