Cover Image

Automatically Benchmarking LLM Code Agents through Agent-Driven Annotation and Evaluation

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Fu, Lingyue, Zhang, Bolun, Guan, Hao, Zhu, Yaoming, Qiu, Lin, Liu, Weiwen, Cao, Xuezhi, Cai, Xunliang, Zhang, Weinan, Yu, Yong
Format:	Preprint
Published:	2025
Subjects:	Software Engineering Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

SWE-Cycle: Benchmarking Code Agents across the Complete Issue Resolution Cycle
by: Guan, Hao, et al.
Published: (2026)

CoreCodeBench: Decoupling Code Intelligence via Fine-Grained Repository-Level Tasks
by: Fu, Lingyue, et al.
Published: (2025)

CATArena: Evaluating Evolutionary Capabilities of Code Agents via Iterative Tournaments
by: Fu, Lingyue, et al.
Published: (2025)

DebugTA: An LLM-Based Agent for Simplifying Debugging and Teaching in Programming Education
by: Fu, Lingyue, et al.
Published: (2025)

AdverMCTS: Combating Pseudo-Correctness in Code Generation via Adversarial Monte Carlo Tree Search
by: Li, Qingyao, et al.
Published: (2026)

RedCodeAgent: Automatic Red-teaming Agent against Diverse Code Agents
by: Guo, Chengquan, et al.
Published: (2025)

RepoTransAgent: Multi-Agent LLM Framework for Repository-Aware Code Translation
by: Guan, Ziqi, et al.
Published: (2025)

Agent-Driven Automatic Software Improvement
by: Ruiz, Fernando Vallecillos
Published: (2024)

Code Review Agent Benchmark
by: Zhang, Yuntong, et al.
Published: (2026)

CodeGRAG: Bridging the Gap between Natural Language and Programming Language via Graphical Retrieval Augmented Generation
by: Du, Kounianhua, et al.
Published: (2024)

SABER: Benchmarking Operational Safety of LLM Coding Agents in Stateful Project Workspaces
by: Hu, Qi, et al.
Published: (2026)

Evaluation-Driven Development and Operations of LLM Agents: A Process Model and Reference Architecture
by: Xia, Boming, et al.
Published: (2024)

GraphCodeAgent: Dual Graph-Guided LLM Agent for Retrieval-Augmented Repo-Level Code Generation
by: Li, Jia, et al.
Published: (2025)

Agent-Diff: Benchmarking LLM Agents on Enterprise API Tasks via Code Execution with State-Diff-Based Evaluation
by: Pysklo, Hubert M., et al.
Published: (2026)

Agentic Harness Engineering: Observability-Driven Automatic Evolution of Coding-Agent Harnesses
by: Lin, Jiahang, et al.
Published: (2026)

Continuous Benchmark Generation for Evaluating Enterprise-scale LLM Agents
by: Saxena, Divyanshu, et al.
Published: (2025)

Automatic Generation of Benchmarks and Reliable LLM Judgment for Code Tasks
by: Farchi, Eitan, et al.
Published: (2024)

Externalization in LLM Agents: A Unified Review of Memory, Skills, Protocols and Harness Engineering
by: Zhou, Chenyu, et al.
Published: (2026)

RiskTagger: An LLM-based Agent for Automatic Annotation of Web3 Crypto Money Laundering Behaviors
by: Lin, Dan, et al.
Published: (2025)

REAP: Automatic Curation of Coding Agent Benchmarks from Interactive Production Usage
by: Jha, Smriti, et al.
Published: (2026)

HumanEvalComm: Benchmarking the Communication Competence of Code Generation for LLMs and LLM Agent
by: Wu, Jie JW, et al.
Published: (2024)

CodeCoR: An LLM-Based Self-Reflective Multi-Agent Framework for Code Generation
by: Pan, Ruwei, et al.
Published: (2025)

Environment-in-the-Loop: Rethinking Code Migration with LLM-based Agents
by: Li, Xiang, et al.
Published: (2026)

DecompileBench: A Comprehensive Benchmark for Evaluating Decompilers in Real-World Scenarios
by: Gao, Zeyu, et al.
Published: (2025)

AgentFM: Role-Aware Failure Management for Distributed Databases with LLM-Driven Multi-Agents
by: Zhang, Lingzhe, et al.
Published: (2025)

PyBench: Evaluating LLM Agent on various real-world coding tasks
by: Zhang, Yaolun, et al.
Published: (2024)

A Survey of Code Review Benchmarks and Evaluation Practices in Pre-LLM and LLM Era
by: Khan, Taufiqul Islam, et al.
Published: (2026)

ResearchEnvBench: Benchmarking Agents on Environment Synthesis for Research Code Execution
by: Wang, Yubang, et al.
Published: (2026)

ISD-Agent-Bench: A Comprehensive Benchmark for Evaluating LLM-based Instructional Design Agents
by: Jeon, YoungHoon, et al.
Published: (2026)

Breaking, Stale, or Missing? Benchmarking Coding Agents on Project-Level Test Evolution
by: Shang, Ye, et al.
Published: (2026)

NoCode-bench: A Benchmark for Evaluating Natural Language-Driven Feature Addition
by: Deng, Le, et al.
Published: (2025)

AI-Generated Smells: An Analysis of Code and Architecture in LLM and Agent-Driven Development
by: Zhu, Yuecai, et al.
Published: (2026)

AutoCodeBench: Large Language Models are Automatic Code Benchmark Generators
by: Chou, Jason, et al.
Published: (2025)

TraceCoder: A Trace-Driven Multi-Agent Framework for Automated Debugging of LLM-Generated Code
by: Huang, Jiangping, et al.
Published: (2026)

ABTest: Behavior-Driven Testing for AI Coding Agents
by: Dai, Wuyang, et al.
Published: (2026)

OmniCode: A Benchmark for Evaluating Software Engineering Agents
by: Sonwane, Atharv, et al.
Published: (2026)

Benchmarking and Studying the LLM-based Agent System in End-to-End Software Development
by: Zeng, Zhengran, et al.
Published: (2025)

LoCoBench-Agent: An Interactive Benchmark for LLM Agents in Long-Context Software Engineering
by: Qiu, Jielin, et al.
Published: (2025)

A Benchmark for Evaluating Repository-Level Code Agents with Intermediate Reasoning on Feature Addition Task
by: Liu, Shuhan, et al.
Published: (2026)

ClarEval: A Benchmark for Evaluating Clarification Skills of Code Agents under Ambiguous Instructions
by: Li, Jialin, et al.
Published: (2026)