Automatically Benchmarking LLM Code Agents through Agent-Driven Annotation and Evaluation
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914414063517696 |
|---|---|
| author | Fu, Lingyue Zhang, Bolun Guan, Hao Zhu, Yaoming Qiu, Lin Liu, Weiwen Cao, Xuezhi Cai, Xunliang Zhang, Weinan Yu, Yong |
| author_facet | Fu, Lingyue Zhang, Bolun Guan, Hao Zhu, Yaoming Qiu, Lin Liu, Weiwen Cao, Xuezhi Cai, Xunliang Zhang, Weinan Yu, Yong |
| contents | Recent advances in code agents have enabled automated software development at the project level, supported by large language models (LLMs). However, existing benchmarks for code agent evaluation face two major limitations. First, creating high-quality project-level evaluation datasets requires extensive domain expertise, leading to prohibitive annotation costs and limited diversity. Second, while recent Agent-as-a-Judge paradigms address the rigidity of traditional unit tests by enabling flexible metrics, their reliance on In-Context Learning (ICL) with general LLMs often results in inaccurate assessments that misalign with human standards. To address these challenges, we propose an agent-driven benchmark construction pipeline that leverages human supervision to efficiently generate diverse project-level tasks. Based on this, we introduce PRDBench, comprising 50 real-world Python projects across 20 domains, each with structured Product Requirement Documents (PRDs) and comprehensive criteria. Furthermore, to overcome the inaccuracy of general LLM judges, we propose a highly reliable evaluation framework powered by a specialized, fine-tuned model. Based on Qwen3-Coder-30B, our dedicated PRDJudge achieves over 90% human alignment in fixed-interface scenarios. Extensive experiments demonstrate that our suite provides a scalable, robust, and highly accurate framework for assessing state-of-the-art code agents. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_24358 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Automatically Benchmarking LLM Code Agents through Agent-Driven Annotation and Evaluation Fu, Lingyue Zhang, Bolun Guan, Hao Zhu, Yaoming Qiu, Lin Liu, Weiwen Cao, Xuezhi Cai, Xunliang Zhang, Weinan Yu, Yong Software Engineering Computation and Language Recent advances in code agents have enabled automated software development at the project level, supported by large language models (LLMs). However, existing benchmarks for code agent evaluation face two major limitations. First, creating high-quality project-level evaluation datasets requires extensive domain expertise, leading to prohibitive annotation costs and limited diversity. Second, while recent Agent-as-a-Judge paradigms address the rigidity of traditional unit tests by enabling flexible metrics, their reliance on In-Context Learning (ICL) with general LLMs often results in inaccurate assessments that misalign with human standards. To address these challenges, we propose an agent-driven benchmark construction pipeline that leverages human supervision to efficiently generate diverse project-level tasks. Based on this, we introduce PRDBench, comprising 50 real-world Python projects across 20 domains, each with structured Product Requirement Documents (PRDs) and comprehensive criteria. Furthermore, to overcome the inaccuracy of general LLM judges, we propose a highly reliable evaluation framework powered by a specialized, fine-tuned model. Based on Qwen3-Coder-30B, our dedicated PRDJudge achieves over 90% human alignment in fixed-interface scenarios. Extensive experiments demonstrate that our suite provides a scalable, robust, and highly accurate framework for assessing state-of-the-art code agents. |
| title | Automatically Benchmarking LLM Code Agents through Agent-Driven Annotation and Evaluation |
| topic | Software Engineering Computation and Language |
| url | https://arxiv.org/abs/2510.24358 |