DevBench: A Realistic, Developer-Informed Benchmark for Code Generation Models
Fuente:
arXiv
Guardado en:
| Autores principales: | Kumarappan, Adarsh, Golnari, Pareesa Ameneh, Wen, Wen, Liu, Xiaoyu, Ryan, Gabriel, Sun, Yuting, Fu, Shengyu, Nallipogu, Elsie |
|---|---|
| Formato: | Preprint |
| Publicado: |
2026
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
Ejemplares similares
Sphinx: Benchmarking and Modeling for LLM-Driven Pull Request Review
por: Zhang, Daoan, et al.
Publicado: (2026)
por: Zhang, Daoan, et al.
Publicado: (2026)
ProjDevBench: Benchmarking AI Coding Agents on End-to-End Project Development
por: Lu, Pengrui, et al.
Publicado: (2026)
por: Lu, Pengrui, et al.
Publicado: (2026)
DSCodeBench: A Realistic Benchmark for Data Science Code Generation
por: Ouyang, Shuyin, et al.
Publicado: (2025)
por: Ouyang, Shuyin, et al.
Publicado: (2025)
MobileDev-Bench: A Benchmark for Issue Resolution in Mobile Application Development
por: Fakorede, Moshood A., et al.
Publicado: (2026)
por: Fakorede, Moshood A., et al.
Publicado: (2026)
DevOps Automation Pipeline Deployment with IaC (Infrastructure as Code)
por: Saxena, Adarsh, et al.
Publicado: (2025)
por: Saxena, Adarsh, et al.
Publicado: (2025)
QuanBench: Benchmarking Quantum Code Generation with Large Language Models
por: Guo, Xiaoyu, et al.
Publicado: (2025)
por: Guo, Xiaoyu, et al.
Publicado: (2025)
Lita: Light Agent Uncovers the Agentic Coding Capabilities of LLMs
por: Dai, Hankun, et al.
Publicado: (2025)
por: Dai, Hankun, et al.
Publicado: (2025)
Skeleton-Guided-Translation: A Benchmarking Framework for Code Repository Translation with Fine-Grained Quality Evaluation
por: Zhang, Xing, et al.
Publicado: (2025)
por: Zhang, Xing, et al.
Publicado: (2025)
SimpleDevQA: Benchmarking Large Language Models on Development Knowledge QA
por: Zhang, Jing, et al.
Publicado: (2025)
por: Zhang, Jing, et al.
Publicado: (2025)
SWE-WebDevBench: Evaluating Coding Agent Application Platforms as Virtual Software Agencies
por: Saxena, Siddhant, et al.
Publicado: (2026)
por: Saxena, Siddhant, et al.
Publicado: (2026)
Embedded DevOps: A Survey on the Application of DevOps Practices in Embedded Software and Firmware Development
por: Katapara, Parthiv, et al.
Publicado: (2025)
por: Katapara, Parthiv, et al.
Publicado: (2025)
SemRep: Generative Code Representation Learning with Code Transformations
por: Li, Weichen, et al.
Publicado: (2026)
por: Li, Weichen, et al.
Publicado: (2026)
Delulu: A Verified Multi-Lingual Benchmark for Code Hallucination Detection in Fill-in-the-Middle Tasks
por: Erfanian, Mahdi, et al.
Publicado: (2026)
por: Erfanian, Mahdi, et al.
Publicado: (2026)
SWE-Bench+: Enhanced Coding Benchmark for LLMs
por: Aleithan, Reem, et al.
Publicado: (2024)
por: Aleithan, Reem, et al.
Publicado: (2024)
GameDevBench: Evaluating Agentic Capabilities Through Game Development
por: Chi, Wayne, et al.
Publicado: (2026)
por: Chi, Wayne, et al.
Publicado: (2026)
DevGPT: Studying Developer-ChatGPT Conversations
por: Xiao, Tao, et al.
Publicado: (2023)
por: Xiao, Tao, et al.
Publicado: (2023)
DevEval: A Manually-Annotated Code Generation Benchmark Aligned with Real-World Code Repositories
por: Li, Jia, et al.
Publicado: (2024)
por: Li, Jia, et al.
Publicado: (2024)
EvoCodeBench: A Human-Performance Benchmark for Self-Evolving LLM-Driven Coding Systems
por: Zhang, Wentao, et al.
Publicado: (2026)
por: Zhang, Wentao, et al.
Publicado: (2026)
FeatureBench: Benchmarking Agentic Coding for Complex Feature Development
por: Zhou, Qixing, et al.
Publicado: (2026)
por: Zhou, Qixing, et al.
Publicado: (2026)
Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfaces
por: Merrill, Mike A., et al.
Publicado: (2026)
por: Merrill, Mike A., et al.
Publicado: (2026)
Towards a Science of Developer eXperience (DevX)
por: Combemale, Benoit
Publicado: (2025)
por: Combemale, Benoit
Publicado: (2025)
Low-Code Paradox in DevOps: Security and Governance Insights from Practitioners
por: Akbar, Muhammad Azeem, et al.
Publicado: (2026)
por: Akbar, Muhammad Azeem, et al.
Publicado: (2026)
RubberDuckBench: A Benchmark for AI Coding Assistants
por: Mohammed, Ferida, et al.
Publicado: (2026)
por: Mohammed, Ferida, et al.
Publicado: (2026)
iReDev: A Knowledge-Driven Multi-Agent Framework for Intelligent Requirements Development
por: Jin, Dongming, et al.
Publicado: (2025)
por: Jin, Dongming, et al.
Publicado: (2025)
RealBench: A Repo-Level Code Generation Benchmark Aligned with Real-World Software Development Practices
por: Li, Jia, et al.
Publicado: (2026)
por: Li, Jia, et al.
Publicado: (2026)
DI-BENCH: Benchmarking Large Language Models on Dependency Inference with Testable Repositories at Scale
por: Zhang, Linghao, et al.
Publicado: (2025)
por: Zhang, Linghao, et al.
Publicado: (2025)
FEA-Bench: A Benchmark for Evaluating Repository-Level Code Generation for Feature Implementation
por: Li, Wei, et al.
Publicado: (2025)
por: Li, Wei, et al.
Publicado: (2025)
Deployability-Centric Infrastructure-as-Code Generation: Fail, Learn, Refine, and Succeed through LLM-Empowered DevOps Simulation
por: Zhang, Tianyi, et al.
Publicado: (2025)
por: Zhang, Tianyi, et al.
Publicado: (2025)
Saving SWE-Bench: A Benchmark Mutation Approach for Realistic Agent Evaluation
por: Garg, Spandan, et al.
Publicado: (2025)
por: Garg, Spandan, et al.
Publicado: (2025)
ML-Dev-Bench: Comparative Analysis of AI Agents on ML development workflows
por: Padigela, Harshith, et al.
Publicado: (2025)
por: Padigela, Harshith, et al.
Publicado: (2025)
AutoDev: Automated AI-Driven Development
por: Tufano, Michele, et al.
Publicado: (2024)
por: Tufano, Michele, et al.
Publicado: (2024)
The road to Sustainable DevOps
por: Herati, Darwish Ahmad, et al.
Publicado: (2025)
por: Herati, Darwish Ahmad, et al.
Publicado: (2025)
DevOps-Gym: Benchmarking AI Agents in Software DevOps Cycle
por: Tang, Yuheng, et al.
Publicado: (2026)
por: Tang, Yuheng, et al.
Publicado: (2026)
HumanEvo: An Evolution-aware Benchmark for More Realistic Evaluation of Repository-level Code Generation
por: Zheng, Dewu, et al.
Publicado: (2024)
por: Zheng, Dewu, et al.
Publicado: (2024)
Bridging Design and Development with Automated Declarative UI Code Generation
por: Zhou, Ting, et al.
Publicado: (2024)
por: Zhou, Ting, et al.
Publicado: (2024)
MigrationBench: Repository-Level Code Migration Benchmark from Java 8
por: Liu, Linbo, et al.
Publicado: (2025)
por: Liu, Linbo, et al.
Publicado: (2025)
OSS-Bench: Benchmark Generator for Coding LLMs
por: Jiang, Yuancheng, et al.
Publicado: (2025)
por: Jiang, Yuancheng, et al.
Publicado: (2025)
SimdBench: Benchmarking Large Language Models for SIMD-Intrinsic Code Generation
por: He, Yibo, et al.
Publicado: (2025)
por: He, Yibo, et al.
Publicado: (2025)
Code2Bench: Scaling Source and Rigor for Dynamic Benchmark Construction
por: Zhang, Zhe, et al.
Publicado: (2025)
por: Zhang, Zhe, et al.
Publicado: (2025)
DevMuT: Testing Deep Learning Framework via Developer Expertise-Based Mutation
por: Mu, Yanzhou, et al.
Publicado: (2025)
por: Mu, Yanzhou, et al.
Publicado: (2025)
Ejemplares similares
-
Sphinx: Benchmarking and Modeling for LLM-Driven Pull Request Review
por: Zhang, Daoan, et al.
Publicado: (2026) -
ProjDevBench: Benchmarking AI Coding Agents on End-to-End Project Development
por: Lu, Pengrui, et al.
Publicado: (2026) -
DSCodeBench: A Realistic Benchmark for Data Science Code Generation
por: Ouyang, Shuyin, et al.
Publicado: (2025) -
MobileDev-Bench: A Benchmark for Issue Resolution in Mobile Application Development
por: Fakorede, Moshood A., et al.
Publicado: (2026) -
DevOps Automation Pipeline Deployment with IaC (Infrastructure as Code)
por: Saxena, Adarsh, et al.
Publicado: (2025)