Rigor, Reliability, and Reproducibility Matter: A Decade-Scale Survey of 572 Code Benchmarks
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866910014814289920 |
|---|---|
| author | Cao, Jialun Chan, Yuk-Kit Ling, Zixuan Wang, Wenxuan Li, Shuqing Liu, Mingwei Qiao, Ruixi Han, Yuting Wang, Chaozheng Yu, Boxi He, Pinjia Wang, Shuai Zheng, Zibin Lyu, Michael R. Cheung, Shing-Chi |
| author_facet | Cao, Jialun Chan, Yuk-Kit Ling, Zixuan Wang, Wenxuan Li, Shuqing Liu, Mingwei Qiao, Ruixi Han, Yuting Wang, Chaozheng Yu, Boxi He, Pinjia Wang, Shuai Zheng, Zibin Lyu, Michael R. Cheung, Shing-Chi |
| contents | Code-related benchmarks play a critical role in evaluating large language models (LLMs), yet their quality fundamentally shapes how the community interprets model capabilities. In the past few years, awareness of benchmark quality has grown. Yet, after a decade-scale (2014-2025) survey over 572 code benchmarks, we observed a lag between growing awareness and actual practice. For example, in 2025 alone, the number of benchmarks that ignore code coverage when providing test cases nearly matches the total count accumulated across the previous ten years. In response, we take a clear position: Code benchmarks must prioritize rigor in benchmark construction, reliability in evaluation, and reproducibility in release. To operationalize this position, we introduce a code benchmark guideline HOW2BENCH with 55 checklists. Finally, our further human study also exposed that the current issues not only stem from the significant effort required, but also from a lack of awareness regarding their importance. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_10711 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Rigor, Reliability, and Reproducibility Matter: A Decade-Scale Survey of 572 Code Benchmarks Cao, Jialun Chan, Yuk-Kit Ling, Zixuan Wang, Wenxuan Li, Shuqing Liu, Mingwei Qiao, Ruixi Han, Yuting Wang, Chaozheng Yu, Boxi He, Pinjia Wang, Shuai Zheng, Zibin Lyu, Michael R. Cheung, Shing-Chi Software Engineering Artificial Intelligence Computation and Language Code-related benchmarks play a critical role in evaluating large language models (LLMs), yet their quality fundamentally shapes how the community interprets model capabilities. In the past few years, awareness of benchmark quality has grown. Yet, after a decade-scale (2014-2025) survey over 572 code benchmarks, we observed a lag between growing awareness and actual practice. For example, in 2025 alone, the number of benchmarks that ignore code coverage when providing test cases nearly matches the total count accumulated across the previous ten years. In response, we take a clear position: Code benchmarks must prioritize rigor in benchmark construction, reliability in evaluation, and reproducibility in release. To operationalize this position, we introduce a code benchmark guideline HOW2BENCH with 55 checklists. Finally, our further human study also exposed that the current issues not only stem from the significant effort required, but also from a lack of awareness regarding their importance. |
| title | Rigor, Reliability, and Reproducibility Matter: A Decade-Scale Survey of 572 Code Benchmarks |
| topic | Software Engineering Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2501.10711 |