Lost in Benchmarks? Rethinking Large Language Model Benchmarking with Item Response Theory
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866908769193033728 |
|---|---|
| author | Zhou, Hongli Huang, Hui Zhao, Ziqing Han, Lvyuan Wang, Huicheng Chen, Kehai Yang, Muyun Bao, Wei Dong, Jian Xu, Bing Zhu, Conghui Cao, Hailong Zhao, Tiejun |
| author_facet | Zhou, Hongli Huang, Hui Zhao, Ziqing Han, Lvyuan Wang, Huicheng Chen, Kehai Yang, Muyun Bao, Wei Dong, Jian Xu, Bing Zhu, Conghui Cao, Hailong Zhao, Tiejun |
| contents | The evaluation of large language models (LLMs) via benchmarks is widespread, yet inconsistencies between different leaderboards and poor separability among top models raise concerns about their ability to accurately reflect authentic model capabilities. This paper provides a critical analysis of benchmark effectiveness, examining mainstream prominent LLM benchmarks using results from diverse models. We first propose Pseudo-Siamese Network for Item Response Theory (PSN-IRT), an enhanced Item Response Theory framework that incorporates a rich set of item parameters within an IRT-grounded architecture. PSN-IRT can be utilized for accurate and reliable estimations of item characteristics and model abilities. Based on PSN-IRT, we conduct extensive analysis on 11 LLM benchmarks comprising 41,871 items, revealing significant and varied shortcomings in their measurement quality. Furthermore, we demonstrate that leveraging PSN-IRT is able to construct smaller benchmarks while maintaining stronger alignment with human preference. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_15055 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Lost in Benchmarks? Rethinking Large Language Model Benchmarking with Item Response Theory Zhou, Hongli Huang, Hui Zhao, Ziqing Han, Lvyuan Wang, Huicheng Chen, Kehai Yang, Muyun Bao, Wei Dong, Jian Xu, Bing Zhu, Conghui Cao, Hailong Zhao, Tiejun Computation and Language The evaluation of large language models (LLMs) via benchmarks is widespread, yet inconsistencies between different leaderboards and poor separability among top models raise concerns about their ability to accurately reflect authentic model capabilities. This paper provides a critical analysis of benchmark effectiveness, examining mainstream prominent LLM benchmarks using results from diverse models. We first propose Pseudo-Siamese Network for Item Response Theory (PSN-IRT), an enhanced Item Response Theory framework that incorporates a rich set of item parameters within an IRT-grounded architecture. PSN-IRT can be utilized for accurate and reliable estimations of item characteristics and model abilities. Based on PSN-IRT, we conduct extensive analysis on 11 LLM benchmarks comprising 41,871 items, revealing significant and varied shortcomings in their measurement quality. Furthermore, we demonstrate that leveraging PSN-IRT is able to construct smaller benchmarks while maintaining stronger alignment with human preference. |
| title | Lost in Benchmarks? Rethinking Large Language Model Benchmarking with Item Response Theory |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2505.15055 |