CentaurEval: Benchmarking Human-in-the-Loop Value in Agentic Coding
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866910244065509376 |
|---|---|
| author | Luo, Hanjun Ni, Chiming Wen, Jiaheng Huang, Zhimu Wang, Yiran Liao, Bingduo Chung, Sylvia Jin, Yingbin Li, Xinfeng Xu, Wenyuan Wang, XiaoFeng Salam, Hanan |
| author_facet | Luo, Hanjun Ni, Chiming Wen, Jiaheng Huang, Zhimu Wang, Yiran Liao, Bingduo Chung, Sylvia Jin, Yingbin Li, Xinfeng Xu, Wenyuan Wang, XiaoFeng Salam, Hanan |
| contents | LLM-powered coding agents are reshaping the development paradigm. However, existing evaluation systems, neither traditional tests for humans nor benchmarks for LLMs, fail to capture this shift, excluding problems that require both human reasoning to guide solutions and AI efficiency for implementation. We introduce CentaurEval, a unified, ecologically valid benchmark for measuring human-in-the-loop value in coding. CentaurEval's core innovation is its "Collaboration-Necessary" problem templates, which are intractable for standalone LLMs or humans, but solvable through effective collaboration. CentaurEval dynamically instantiates tasks from 45 templates, providing a standardized IDE for humans and a reproducible 450-task toolkit for LLMs. We benchmark 45 participants against 5 LLMs under 4 levels of human intervention. Results show that while LLMs or humans alone achieve poor pass rates (0.67% and 18.89%), human-AI collaboration significantly improves to 31.11%. Our analysis reveals an emerging co-reasoning partnership, challenging the traditional human-tool hierarchy by showing that strategic breakthroughs can originate from either humans or AI. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_04111 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | CentaurEval: Benchmarking Human-in-the-Loop Value in Agentic Coding Luo, Hanjun Ni, Chiming Wen, Jiaheng Huang, Zhimu Wang, Yiran Liao, Bingduo Chung, Sylvia Jin, Yingbin Li, Xinfeng Xu, Wenyuan Wang, XiaoFeng Salam, Hanan Software Engineering Artificial Intelligence Human-Computer Interaction LLM-powered coding agents are reshaping the development paradigm. However, existing evaluation systems, neither traditional tests for humans nor benchmarks for LLMs, fail to capture this shift, excluding problems that require both human reasoning to guide solutions and AI efficiency for implementation. We introduce CentaurEval, a unified, ecologically valid benchmark for measuring human-in-the-loop value in coding. CentaurEval's core innovation is its "Collaboration-Necessary" problem templates, which are intractable for standalone LLMs or humans, but solvable through effective collaboration. CentaurEval dynamically instantiates tasks from 45 templates, providing a standardized IDE for humans and a reproducible 450-task toolkit for LLMs. We benchmark 45 participants against 5 LLMs under 4 levels of human intervention. Results show that while LLMs or humans alone achieve poor pass rates (0.67% and 18.89%), human-AI collaboration significantly improves to 31.11%. Our analysis reveals an emerging co-reasoning partnership, challenging the traditional human-tool hierarchy by showing that strategic breakthroughs can originate from either humans or AI. |
| title | CentaurEval: Benchmarking Human-in-the-Loop Value in Agentic Coding |
| topic | Software Engineering Artificial Intelligence Human-Computer Interaction |
| url | https://arxiv.org/abs/2512.04111 |