Cover Image

REAP: Automatic Curation of Coding Agent Benchmarks from Interactive Production Usage

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Jha, Smriti, Paltenghi, Matteo, Maddila, Chandra, Murali, Vijayaraghavan, Ugare, Shubham, Chandra, Satish
Format:	Preprint
Published:	2026
Subjects:	Software Engineering Artificial Intelligence Machine Learning
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Wink: Recovering from Misbehaviors in Coding Agents
by: Nanda, Rahul, et al.
Published: (2026)

Agentic Code Reasoning
by: Ugare, Shubham, et al.
Published: (2026)

AI-assisted Code Authoring at Scale: Fine-tuning, deploying, and mixed methods evaluation
by: Murali, Vijayaraghavan, et al.
Published: (2023)

Multi-line AI-assisted Code Authoring
by: Dunay, Omer, et al.
Published: (2024)

AI-Assisted Fixes to Code Review Comments at Scale
by: Maddila, Chandra, et al.
Published: (2025)

Follow-up Attention: An Empirical Study of Developer and Neural Model Code Exploration
by: Paltenghi, Matteo, et al.
Published: (2022)

Evaluating Agent-based Program Repair at Google
by: Rondon, Pat, et al.
Published: (2025)

CRQBench: A Benchmark of Code Reasoning Questions
by: Dinella, Elizabeth, et al.
Published: (2024)

Reading Between the Lines: Scalable User Feedback via Implicit Sentiment in Developer Prompts
by: Nam, Daye, et al.
Published: (2025)

AndroidControl-Curated: Revealing the True Potential of GUI Agents through Benchmark Purification
by: Leung, Ho Fai, et al.
Published: (2025)

AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agents
by: Trivedi, Harsh, et al.
Published: (2024)

Code Review Agent Benchmark
by: Zhang, Yuntong, et al.
Published: (2026)

CODESTRUCT: Code Agents over Structured Action Spaces
by: Kim, Myeongsoo, et al.
Published: (2026)

Can Coding Agents Be General Agents?
by: Ivanov, Maksim, et al.
Published: (2026)

CLEVER: A Curated Benchmark for Formally Verified Code Generation
by: Thakur, Amitayush, et al.
Published: (2025)

RedCode: Risky Code Execution and Generation Benchmark for Code Agents
by: Guo, Chengquan, et al.
Published: (2024)

SpecAgent: A Speculative Retrieval and Forecasting Agent for Code Completion
by: Ma, George, et al.
Published: (2025)

miniCodeProps: a Minimal Benchmark for Proving Code Properties
by: Lohn, Evan, et al.
Published: (2024)

Scaling Down to Scale Up: A Cost-Benefit Analysis of Replacing OpenAI's LLM with Open Source SLMs in Production
by: Irugalbandara, Chandra, et al.
Published: (2023)

CRUXEval: A Benchmark for Code Reasoning, Understanding and Execution
by: Gu, Alex, et al.
Published: (2024)

Agentic Bug Reproduction for Effective Automated Program Repair at Google
by: Cheng, Runxiang, et al.
Published: (2025)

LibEvolutionEval: A Benchmark and Study for Version-Specific Code Generation
by: Kuhar, Sachit, et al.
Published: (2024)

Benchmarking Multimodal LLMs on Code Generation for Complex Interactive Webpages
by: Wu, Fan, et al.
Published: (2026)

Automating Code Adaptation for MLOps -- A Benchmarking Study on LLMs
by: Patel, Harsh, et al.
Published: (2024)

Deep-Bench: Deep Learning Benchmark Dataset for Code Generation
by: Daghighfarsoodeh, Alireza, et al.
Published: (2025)

Drawing Pandas: A Benchmark for LLMs in Generating Plotting Code
by: Galimzyanov, Timur, et al.
Published: (2024)

SWE-Bench-CL: Continual Learning for Coding Agents
by: Joshi, Thomas, et al.
Published: (2025)

Automated Cloud Infrastructure-as-Code Reconciliation with AI Agents
by: Yang, Zhenning, et al.
Published: (2025)

CodeGeeX: A Pre-Trained Model for Code Generation with Multilingual Benchmarking on HumanEval-X
by: Zheng, Qinkai, et al.
Published: (2023)

Moving Faster and Reducing Risk: Using LLMs in Release Deployment
by: Abreu, Rui, et al.
Published: (2024)

Benchmarking Reward Hack Detection in Code Environments via Contrastive Analysis
by: Deshpande, Darshan, et al.
Published: (2026)

VeriContest: A Competitive-Programming Benchmark for Verifiable Code Generation
by: Xie, Zichen, et al.
Published: (2026)

DialogAgent: An Auto-engagement Agent for Code Question Answering Data Production
by: Liang, Xiaoyun, et al.
Published: (2024)

DevBench: A Realistic, Developer-Informed Benchmark for Code Generation Models
by: Kumarappan, Adarsh, et al.
Published: (2026)

How Efficient is LLM-Generated Code? A Rigorous & High-Standard Benchmark
by: Qiu, Ruizhong, et al.
Published: (2024)

Automatic Detection of LLM-Generated Code: A Comparative Case Study of Contemporary Models Across Function and Class Granularities
by: Rahman, Musfiqur, et al.
Published: (2024)

AutoSafeCoder: A Multi-Agent Framework for Securing LLM Code Generation through Static Analysis and Fuzz Testing
by: Nunez, Ana, et al.
Published: (2024)

Your Code Agent Can Grow Alongside You with Structured Memory
by: Deng, Yi-Xuan, et al.
Published: (2026)

Prism: Dynamic and Flexible Benchmarking of LLMs Code Generation with Monte Carlo Tree Search
by: Majdinasab, Vahid, et al.
Published: (2025)

Beyond Output Correctness: Benchmarking and Evaluating Large Language Model Reasoning in Coding Tasks
by: Li, Yuangang, et al.
Published: (2026)