Saved in:
| Main Authors: | , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2510.10457 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912644081909760 |
|---|---|
| author | Wang, Shaobo Wang, Cong Fu, Wenjie Min, Yue Feng, Mingquan Guan, Isabel Hu, Xuming He, Conghui Wang, Cunxiang Yang, Kexin Ren, Xingzhang Huang, Fei Liu, Dayiheng Zhang, Linfeng |
| author_facet | Wang, Shaobo Wang, Cong Fu, Wenjie Min, Yue Feng, Mingquan Guan, Isabel Hu, Xuming He, Conghui Wang, Cunxiang Yang, Kexin Ren, Xingzhang Huang, Fei Liu, Dayiheng Zhang, Linfeng |
| contents | As the demand for comprehensive evaluations of diverse model capabilities steadily increases, benchmark suites have correspondingly grown significantly in scale. Despite notable advances in redundancy reduction and subset-level performance prediction, a systematic framework that effectively integrates these methods to ensure both prediction accuracy and ranking consistency is still largely elusive. In this paper, we first perform a sample-level analysis of benchmark redundancy and identify several highly similar samples that can be eliminated. Besides, we frame benchmark compression as an optimization problem with the aim of score reconstruction. Building on these, we then propose EssenceBench, a coarse-to-fine framework utilizing an iterative Genetic Algorithm (GA), which takes the advantages of fitness-based subset search and attribution-based sample search. Compared to previous methods, our approach yields superior compression results with lower reconstruction error and markedly higher efficiency. In particular, on the HellaSwag benchmark (10K samples), our method preserves the ranking of all models shifting within 5% using 25x fewer samples, and achieves 95% ranking preservation shifting within 5% using only 200x fewer samples. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_10457 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Rethinking LLM Evaluation: Can We Evaluate LLMs with 200x Less Data? Wang, Shaobo Wang, Cong Fu, Wenjie Min, Yue Feng, Mingquan Guan, Isabel Hu, Xuming He, Conghui Wang, Cunxiang Yang, Kexin Ren, Xingzhang Huang, Fei Liu, Dayiheng Zhang, Linfeng Computation and Language Machine Learning As the demand for comprehensive evaluations of diverse model capabilities steadily increases, benchmark suites have correspondingly grown significantly in scale. Despite notable advances in redundancy reduction and subset-level performance prediction, a systematic framework that effectively integrates these methods to ensure both prediction accuracy and ranking consistency is still largely elusive. In this paper, we first perform a sample-level analysis of benchmark redundancy and identify several highly similar samples that can be eliminated. Besides, we frame benchmark compression as an optimization problem with the aim of score reconstruction. Building on these, we then propose EssenceBench, a coarse-to-fine framework utilizing an iterative Genetic Algorithm (GA), which takes the advantages of fitness-based subset search and attribution-based sample search. Compared to previous methods, our approach yields superior compression results with lower reconstruction error and markedly higher efficiency. In particular, on the HellaSwag benchmark (10K samples), our method preserves the ranking of all models shifting within 5% using 25x fewer samples, and achieves 95% ranking preservation shifting within 5% using only 200x fewer samples. |
| title | Rethinking LLM Evaluation: Can We Evaluate LLMs with 200x Less Data? |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2510.10457 |