Salvato in:
| Autori principali: | , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2026
|
| Soggetti: | |
| Accesso online: | https://arxiv.org/abs/2603.01051 |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866918363928723456 |
|---|---|
| author | Atif, Mohammad Chopra, Kriti Tsai, Fang-Ying Kilic, Ozgur O. Wang, Tianle Dong, Zhihua Benjamin, Douglas Leggett, Charles Lin, Meifeng Calafiura, Paolo Habib, Salman |
| author_facet | Atif, Mohammad Chopra, Kriti Tsai, Fang-Ying Kilic, Ozgur O. Wang, Tianle Dong, Zhihua Benjamin, Douglas Leggett, Charles Lin, Meifeng Calafiura, Paolo Habib, Salman |
| contents | Large Language Models (LLM) are increasingly used for software development, yet existing benchmarks for LLM-based coding assistance do not reflect the constraints of High Energy Physics (HEP) and High Performance Computing (HPC) software. Code correctness must respect science constraints and changes must integrate into large, performance-critical codebases with complex dependencies and build systems. The primary contribution of this paper is the development of practical, repeatable benchmarks that quantify LLM performance on HEP/HPC-relevant tasks. We introduce three evaluation tracks -- code documentation benchmarks measure the ability of an LLM to generate Doxygen-style comments, code generation benchmarks evaluate end-to-end usability on representative GPU kernels, and graphical data analysis benchmarks evaluate vision-enabled LLMs. These benchmarks provide a unified framework for measuring progress in scientific coding assistance across documentation quality, code generation robustness, and multimodal validation analysis. By emphasizing repeatability, automated scoring, and domain-relevant failure modes, the suite enables fair comparisons of models and settings while supporting future work on methods that improve reliability for HEP/HPC software development. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_01051 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | CelloAI Benchmarks: Toward Repeatable Evaluation of AI Assistants Atif, Mohammad Chopra, Kriti Tsai, Fang-Ying Kilic, Ozgur O. Wang, Tianle Dong, Zhihua Benjamin, Douglas Leggett, Charles Lin, Meifeng Calafiura, Paolo Habib, Salman High Energy Physics - Experiment Software Engineering Large Language Models (LLM) are increasingly used for software development, yet existing benchmarks for LLM-based coding assistance do not reflect the constraints of High Energy Physics (HEP) and High Performance Computing (HPC) software. Code correctness must respect science constraints and changes must integrate into large, performance-critical codebases with complex dependencies and build systems. The primary contribution of this paper is the development of practical, repeatable benchmarks that quantify LLM performance on HEP/HPC-relevant tasks. We introduce three evaluation tracks -- code documentation benchmarks measure the ability of an LLM to generate Doxygen-style comments, code generation benchmarks evaluate end-to-end usability on representative GPU kernels, and graphical data analysis benchmarks evaluate vision-enabled LLMs. These benchmarks provide a unified framework for measuring progress in scientific coding assistance across documentation quality, code generation robustness, and multimodal validation analysis. By emphasizing repeatability, automated scoring, and domain-relevant failure modes, the suite enables fair comparisons of models and settings while supporting future work on methods that improve reliability for HEP/HPC software development. |
| title | CelloAI Benchmarks: Toward Repeatable Evaluation of AI Assistants |
| topic | High Energy Physics - Experiment Software Engineering |
| url | https://arxiv.org/abs/2603.01051 |