Imagen de Portada

DevBench: A Realistic, Developer-Informed Benchmark for Code Generation Models

Fuente: arXiv

Guardado en:

Detalles Bibliográficos
Autores principales:	Kumarappan, Adarsh, Golnari, Pareesa Ameneh, Wen, Wen, Liu, Xiaoyu, Ryan, Gabriel, Sun, Yuting, Fu, Shengyu, Nallipogu, Elsie
Formato:	Preprint
Publicado:	2026
Materias:	Machine Learning Artificial Intelligence Software Engineering
Acceso en línea:	Acceder al recurso
Etiquetas:	Agregar Etiqueta Sin Etiquetas, Sea el primero en etiquetar este registro!

Ejemplares similares

Sphinx: Benchmarking and Modeling for LLM-Driven Pull Request Review
por: Zhang, Daoan, et al.
Publicado: (2026)

ProjDevBench: Benchmarking AI Coding Agents on End-to-End Project Development
por: Lu, Pengrui, et al.
Publicado: (2026)

DSCodeBench: A Realistic Benchmark for Data Science Code Generation
por: Ouyang, Shuyin, et al.
Publicado: (2025)

MobileDev-Bench: A Benchmark for Issue Resolution in Mobile Application Development
por: Fakorede, Moshood A., et al.
Publicado: (2026)

DevOps Automation Pipeline Deployment with IaC (Infrastructure as Code)
por: Saxena, Adarsh, et al.
Publicado: (2025)

QuanBench: Benchmarking Quantum Code Generation with Large Language Models
por: Guo, Xiaoyu, et al.
Publicado: (2025)

Lita: Light Agent Uncovers the Agentic Coding Capabilities of LLMs
por: Dai, Hankun, et al.
Publicado: (2025)

Skeleton-Guided-Translation: A Benchmarking Framework for Code Repository Translation with Fine-Grained Quality Evaluation
por: Zhang, Xing, et al.
Publicado: (2025)

SimpleDevQA: Benchmarking Large Language Models on Development Knowledge QA
por: Zhang, Jing, et al.
Publicado: (2025)

SWE-WebDevBench: Evaluating Coding Agent Application Platforms as Virtual Software Agencies
por: Saxena, Siddhant, et al.
Publicado: (2026)

Embedded DevOps: A Survey on the Application of DevOps Practices in Embedded Software and Firmware Development
por: Katapara, Parthiv, et al.
Publicado: (2025)

SemRep: Generative Code Representation Learning with Code Transformations
por: Li, Weichen, et al.
Publicado: (2026)

Delulu: A Verified Multi-Lingual Benchmark for Code Hallucination Detection in Fill-in-the-Middle Tasks
por: Erfanian, Mahdi, et al.
Publicado: (2026)

SWE-Bench+: Enhanced Coding Benchmark for LLMs
por: Aleithan, Reem, et al.
Publicado: (2024)

GameDevBench: Evaluating Agentic Capabilities Through Game Development
por: Chi, Wayne, et al.
Publicado: (2026)

DevGPT: Studying Developer-ChatGPT Conversations
por: Xiao, Tao, et al.
Publicado: (2023)

DevEval: A Manually-Annotated Code Generation Benchmark Aligned with Real-World Code Repositories
por: Li, Jia, et al.
Publicado: (2024)

EvoCodeBench: A Human-Performance Benchmark for Self-Evolving LLM-Driven Coding Systems
por: Zhang, Wentao, et al.
Publicado: (2026)

FeatureBench: Benchmarking Agentic Coding for Complex Feature Development
por: Zhou, Qixing, et al.
Publicado: (2026)

Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces
por: Merrill, Mike A., et al.
Publicado: (2026)

Towards a Science of Developer eXperience (DevX)
por: Combemale, Benoit
Publicado: (2025)

Low-Code Paradox in DevOps: Security and Governance Insights from Practitioners
por: Akbar, Muhammad Azeem, et al.
Publicado: (2026)

RubberDuckBench: A Benchmark for AI Coding Assistants
por: Mohammed, Ferida, et al.
Publicado: (2026)

iReDev: A Knowledge-Driven Multi-Agent Framework for Intelligent Requirements Development
por: Jin, Dongming, et al.
Publicado: (2025)

RealBench: A Repo-Level Code Generation Benchmark Aligned with Real-World Software Development Practices
por: Li, Jia, et al.
Publicado: (2026)

DI-BENCH: Benchmarking Large Language Models on Dependency Inference with Testable Repositories at Scale
por: Zhang, Linghao, et al.
Publicado: (2025)

FEA-Bench: A Benchmark for Evaluating Repository-Level Code Generation for Feature Implementation
por: Li, Wei, et al.
Publicado: (2025)

Deployability-Centric Infrastructure-as-Code Generation: Fail, Learn, Refine, and Succeed through LLM-Empowered DevOps Simulation
por: Zhang, Tianyi, et al.
Publicado: (2025)

Saving SWE-Bench: A Benchmark Mutation Approach for Realistic Agent Evaluation
por: Garg, Spandan, et al.
Publicado: (2025)

ML-Dev-Bench: Comparative Analysis of AI Agents on ML development workflows
por: Padigela, Harshith, et al.
Publicado: (2025)

AutoDev: Automated AI-Driven Development
por: Tufano, Michele, et al.
Publicado: (2024)

The road to Sustainable DevOps
por: Herati, Darwish Ahmad, et al.
Publicado: (2025)

DevOps-Gym: Benchmarking AI Agents in Software DevOps Cycle
por: Tang, Yuheng, et al.
Publicado: (2026)

HumanEvo: An Evolution-aware Benchmark for More Realistic Evaluation of Repository-level Code Generation
por: Zheng, Dewu, et al.
Publicado: (2024)

Bridging Design and Development with Automated Declarative UI Code Generation
por: Zhou, Ting, et al.
Publicado: (2024)

MigrationBench: Repository-Level Code Migration Benchmark from Java 8
por: Liu, Linbo, et al.
Publicado: (2025)

OSS-Bench: Benchmark Generator for Coding LLMs
por: Jiang, Yuancheng, et al.
Publicado: (2025)

SimdBench: Benchmarking Large Language Models for SIMD-Intrinsic Code Generation
por: He, Yibo, et al.
Publicado: (2025)

Code2Bench: Scaling Source and Rigor for Dynamic Benchmark Construction
por: Zhang, Zhe, et al.
Publicado: (2025)

DevMuT: Testing Deep Learning Framework via Developer Expertise-Based Mutation
por: Mu, Yanzhou, et al.
Publicado: (2025)