Towards Comprehensive Benchmarking Infrastructure for LLMs In Software Engineering
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2026
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866912857741852672 |
|---|---|
| author | Rodriguez-Cardenas, Daniel Li, Xiaochang Macedo, Marcos Mastropaolo, Antonio Khati, Dipin Tian, Yuan Shao, Huajie Poshyvanyk, Denys |
| author_facet | Rodriguez-Cardenas, Daniel Li, Xiaochang Macedo, Marcos Mastropaolo, Antonio Khati, Dipin Tian, Yuan Shao, Huajie Poshyvanyk, Denys |
| contents | Large language models for code are advancing fast, yet our ability to evaluate them lags behind. Current benchmarks focus on narrow tasks and single metrics, which hide critical gaps in robustness, interpretability, fairness, efficiency, and real-world usability. They also suffer from inconsistent data engineering practices, limited software engineering context, and widespread contamination issues. To understand these problems and chart a path forward, we combined an in-depth survey of existing benchmarks with insights gathered from a dedicated community workshop. We identified three core barriers to reliable evaluation: the absence of software-engineering-rich datasets, overreliance on ML-centric metrics, and the lack of standardized, reproducible data pipelines. Building on these findings, we introduce BEHELM, a holistic benchmarking infrastructure that unifies software-scenario specification with multi-metric evaluation. BEHELM provides a structured way to assess models across tasks, languages, input and output granularities, and key quality dimensions. Our goal is to reduce the overhead currently required to construct benchmarks while enabling a fair, realistic, and future-proof assessment of LLMs in software engineering. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_21070 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Towards Comprehensive Benchmarking Infrastructure for LLMs In Software Engineering Rodriguez-Cardenas, Daniel Li, Xiaochang Macedo, Marcos Mastropaolo, Antonio Khati, Dipin Tian, Yuan Shao, Huajie Poshyvanyk, Denys Software Engineering Artificial Intelligence Large language models for code are advancing fast, yet our ability to evaluate them lags behind. Current benchmarks focus on narrow tasks and single metrics, which hide critical gaps in robustness, interpretability, fairness, efficiency, and real-world usability. They also suffer from inconsistent data engineering practices, limited software engineering context, and widespread contamination issues. To understand these problems and chart a path forward, we combined an in-depth survey of existing benchmarks with insights gathered from a dedicated community workshop. We identified three core barriers to reliable evaluation: the absence of software-engineering-rich datasets, overreliance on ML-centric metrics, and the lack of standardized, reproducible data pipelines. Building on these findings, we introduce BEHELM, a holistic benchmarking infrastructure that unifies software-scenario specification with multi-metric evaluation. BEHELM provides a structured way to assess models across tasks, languages, input and output granularities, and key quality dimensions. Our goal is to reduce the overhead currently required to construct benchmarks while enabling a fair, realistic, and future-proof assessment of LLMs in software engineering. |
| title | Towards Comprehensive Benchmarking Infrastructure for LLMs In Software Engineering |
| topic | Software Engineering Artificial Intelligence |
| url | https://arxiv.org/abs/2601.21070 |