Cover Image

DevBench: A Realistic, Developer-Informed Benchmark for Code Generation Models

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Kumarappan, Adarsh, Golnari, Pareesa Ameneh, Wen, Wen, Liu, Xiaoyu, Ryan, Gabriel, Sun, Yuting, Fu, Shengyu, Nallipogu, Elsie
Format:	Preprint
Published:	2026
Subjects:	Machine Learning Artificial Intelligence Software Engineering
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Sphinx: Benchmarking and Modeling for LLM-Driven Pull Request Review
by: Zhang, Daoan, et al.
Published: (2026)

ProjDevBench: Benchmarking AI Coding Agents on End-to-End Project Development
by: Lu, Pengrui, et al.
Published: (2026)

DSCodeBench: A Realistic Benchmark for Data Science Code Generation
by: Ouyang, Shuyin, et al.
Published: (2025)

MobileDev-Bench: A Benchmark for Issue Resolution in Mobile Application Development
by: Fakorede, Moshood A., et al.
Published: (2026)

DevOps Automation Pipeline Deployment with IaC (Infrastructure as Code)
by: Saxena, Adarsh, et al.
Published: (2025)

QuanBench: Benchmarking Quantum Code Generation with Large Language Models
by: Guo, Xiaoyu, et al.
Published: (2025)

Lita: Light Agent Uncovers the Agentic Coding Capabilities of LLMs
by: Dai, Hankun, et al.
Published: (2025)

Skeleton-Guided-Translation: A Benchmarking Framework for Code Repository Translation with Fine-Grained Quality Evaluation
by: Zhang, Xing, et al.
Published: (2025)

SimpleDevQA: Benchmarking Large Language Models on Development Knowledge QA
by: Zhang, Jing, et al.
Published: (2025)

SWE-WebDevBench: Evaluating Coding Agent Application Platforms as Virtual Software Agencies
by: Saxena, Siddhant, et al.
Published: (2026)

Embedded DevOps: A Survey on the Application of DevOps Practices in Embedded Software and Firmware Development
by: Katapara, Parthiv, et al.
Published: (2025)

SemRep: Generative Code Representation Learning with Code Transformations
by: Li, Weichen, et al.
Published: (2026)

Delulu: A Verified Multi-Lingual Benchmark for Code Hallucination Detection in Fill-in-the-Middle Tasks
by: Erfanian, Mahdi, et al.
Published: (2026)

SWE-Bench+: Enhanced Coding Benchmark for LLMs
by: Aleithan, Reem, et al.
Published: (2024)

GameDevBench: Evaluating Agentic Capabilities Through Game Development
by: Chi, Wayne, et al.
Published: (2026)

DevGPT: Studying Developer-ChatGPT Conversations
by: Xiao, Tao, et al.
Published: (2023)

DevEval: A Manually-Annotated Code Generation Benchmark Aligned with Real-World Code Repositories
by: Li, Jia, et al.
Published: (2024)

EvoCodeBench: A Human-Performance Benchmark for Self-Evolving LLM-Driven Coding Systems
by: Zhang, Wentao, et al.
Published: (2026)

FeatureBench: Benchmarking Agentic Coding for Complex Feature Development
by: Zhou, Qixing, et al.
Published: (2026)

Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces
by: Merrill, Mike A., et al.
Published: (2026)

Towards a Science of Developer eXperience (DevX)
by: Combemale, Benoit
Published: (2025)

Low-Code Paradox in DevOps: Security and Governance Insights from Practitioners
by: Akbar, Muhammad Azeem, et al.
Published: (2026)

RubberDuckBench: A Benchmark for AI Coding Assistants
by: Mohammed, Ferida, et al.
Published: (2026)

iReDev: A Knowledge-Driven Multi-Agent Framework for Intelligent Requirements Development
by: Jin, Dongming, et al.
Published: (2025)

RealBench: A Repo-Level Code Generation Benchmark Aligned with Real-World Software Development Practices
by: Li, Jia, et al.
Published: (2026)

DI-BENCH: Benchmarking Large Language Models on Dependency Inference with Testable Repositories at Scale
by: Zhang, Linghao, et al.
Published: (2025)

FEA-Bench: A Benchmark for Evaluating Repository-Level Code Generation for Feature Implementation
by: Li, Wei, et al.
Published: (2025)

Deployability-Centric Infrastructure-as-Code Generation: Fail, Learn, Refine, and Succeed through LLM-Empowered DevOps Simulation
by: Zhang, Tianyi, et al.
Published: (2025)

Saving SWE-Bench: A Benchmark Mutation Approach for Realistic Agent Evaluation
by: Garg, Spandan, et al.
Published: (2025)

ML-Dev-Bench: Comparative Analysis of AI Agents on ML development workflows
by: Padigela, Harshith, et al.
Published: (2025)

AutoDev: Automated AI-Driven Development
by: Tufano, Michele, et al.
Published: (2024)

The road to Sustainable DevOps
by: Herati, Darwish Ahmad, et al.
Published: (2025)

DevOps-Gym: Benchmarking AI Agents in Software DevOps Cycle
by: Tang, Yuheng, et al.
Published: (2026)

HumanEvo: An Evolution-aware Benchmark for More Realistic Evaluation of Repository-level Code Generation
by: Zheng, Dewu, et al.
Published: (2024)

Bridging Design and Development with Automated Declarative UI Code Generation
by: Zhou, Ting, et al.
Published: (2024)

MigrationBench: Repository-Level Code Migration Benchmark from Java 8
by: Liu, Linbo, et al.
Published: (2025)

OSS-Bench: Benchmark Generator for Coding LLMs
by: Jiang, Yuancheng, et al.
Published: (2025)

SimdBench: Benchmarking Large Language Models for SIMD-Intrinsic Code Generation
by: He, Yibo, et al.
Published: (2025)

Code2Bench: Scaling Source and Rigor for Dynamic Benchmark Construction
by: Zhang, Zhe, et al.
Published: (2025)

DevMuT: Testing Deep Learning Framework via Developer Expertise-Based Mutation
by: Mu, Yanzhou, et al.
Published: (2025)