ContextBench: A Benchmark for Context Retrieval in Coding Agents
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2026
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866918331769946112 |
|---|---|
| author | Li, Han Zhu, Letian Zhang, Bohan Feng, Rili Wang, Jiaming Pan, Yue Barr, Earl T. Sarro, Federica Chu, Zhaoyang Ye, He |
| author_facet | Li, Han Zhu, Letian Zhang, Bohan Feng, Rili Wang, Jiaming Pan, Yue Barr, Earl T. Sarro, Federica Chu, Zhaoyang Ye, He |
| contents | LLM-based coding agents have shown strong performance on automated issue resolution benchmarks, yet existing evaluations largely focus on final task success, providing limited insight into how agents retrieve and use code context during problem solving. We introduce ContextBench, a process-oriented evaluation of context retrieval in coding agents. ContextBench consists of 1,136 issue-resolution tasks from 66 repositories across eight programming languages, each augmented with human-annotated gold contexts. We further implement an automated evaluation framework that tracks agent trajectories and measures context recall, precision, and efficiency throughout issue resolution. Using ContextBench, we evaluate four frontier LLMs and five coding agents. Our results show that sophisticated agent scaffolding yields only marginal gains in context retrieval ("The Bitter Lesson" of coding agents), LLMs consistently favor recall over precision, and substantial gaps exist between explored and utilized context. ContextBench augments existing end-to-end benchmarks with intermediate gold-context metrics that unbox the issue-resolution process. These contexts offer valuable intermediate signals for guiding LLM reasoning in software tasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_05892 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | ContextBench: A Benchmark for Context Retrieval in Coding Agents Li, Han Zhu, Letian Zhang, Bohan Feng, Rili Wang, Jiaming Pan, Yue Barr, Earl T. Sarro, Federica Chu, Zhaoyang Ye, He Machine Learning D.2.5; I.2.2; I.2.7 LLM-based coding agents have shown strong performance on automated issue resolution benchmarks, yet existing evaluations largely focus on final task success, providing limited insight into how agents retrieve and use code context during problem solving. We introduce ContextBench, a process-oriented evaluation of context retrieval in coding agents. ContextBench consists of 1,136 issue-resolution tasks from 66 repositories across eight programming languages, each augmented with human-annotated gold contexts. We further implement an automated evaluation framework that tracks agent trajectories and measures context recall, precision, and efficiency throughout issue resolution. Using ContextBench, we evaluate four frontier LLMs and five coding agents. Our results show that sophisticated agent scaffolding yields only marginal gains in context retrieval ("The Bitter Lesson" of coding agents), LLMs consistently favor recall over precision, and substantial gaps exist between explored and utilized context. ContextBench augments existing end-to-end benchmarks with intermediate gold-context metrics that unbox the issue-resolution process. These contexts offer valuable intermediate signals for guiding LLM reasoning in software tasks. |
| title | ContextBench: A Benchmark for Context Retrieval in Coding Agents |
| topic | Machine Learning D.2.5; I.2.2; I.2.7 |
| url | https://arxiv.org/abs/2602.05892 |