Benchmark^2: Systematic Evaluation of LLM Benchmarks
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909983704088576 |
|---|---|
| author | Qian, Qi Huang, Chengsong Xu, Jingwen Lv, Changze Wu, Muling Liu, Wenhao Wang, Xiaohua Wang, Zhenghua Huang, Zisu Tian, Muzhao Xu, Jianhan Hu, Kun Wang, He-Da Hu, Yao Huang, Xuanjing Zheng, Xiaoqing |
| author_facet | Qian, Qi Huang, Chengsong Xu, Jingwen Lv, Changze Wu, Muling Liu, Wenhao Wang, Xiaohua Wang, Zhenghua Huang, Zisu Tian, Muzhao Xu, Jianhan Hu, Kun Wang, He-Da Hu, Yao Huang, Xuanjing Zheng, Xiaoqing |
| contents | The rapid proliferation of benchmarks for evaluating large language models (LLMs) has created an urgent need for systematic methods to assess benchmark quality itself. We propose Benchmark^2, a comprehensive framework comprising three complementary metrics: (1) Cross-Benchmark Ranking Consistency, measuring whether a benchmark produces model rankings aligned with peer benchmarks; (2) Discriminability Score, quantifying a benchmark's ability to differentiate between models; and (3) Capability Alignment Deviation, identifying problematic instances where stronger models fail but weaker models succeed within the same model family. We conduct extensive experiments across 15 benchmarks spanning mathematics, reasoning, and knowledge domains, evaluating 11 LLMs across four model families. Our analysis reveals significant quality variations among existing benchmarks and demonstrates that selective benchmark construction based on our metrics can achieve comparable evaluation performance with substantially reduced test sets. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_03986 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Benchmark^2: Systematic Evaluation of LLM Benchmarks Qian, Qi Huang, Chengsong Xu, Jingwen Lv, Changze Wu, Muling Liu, Wenhao Wang, Xiaohua Wang, Zhenghua Huang, Zisu Tian, Muzhao Xu, Jianhan Hu, Kun Wang, He-Da Hu, Yao Huang, Xuanjing Zheng, Xiaoqing Computation and Language The rapid proliferation of benchmarks for evaluating large language models (LLMs) has created an urgent need for systematic methods to assess benchmark quality itself. We propose Benchmark^2, a comprehensive framework comprising three complementary metrics: (1) Cross-Benchmark Ranking Consistency, measuring whether a benchmark produces model rankings aligned with peer benchmarks; (2) Discriminability Score, quantifying a benchmark's ability to differentiate between models; and (3) Capability Alignment Deviation, identifying problematic instances where stronger models fail but weaker models succeed within the same model family. We conduct extensive experiments across 15 benchmarks spanning mathematics, reasoning, and knowledge domains, evaluating 11 LLMs across four model families. Our analysis reveals significant quality variations among existing benchmarks and demonstrates that selective benchmark construction based on our metrics can achieve comparable evaluation performance with substantially reduced test sets. |
| title | Benchmark^2: Systematic Evaluation of LLM Benchmarks |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2601.03986 |