Cover Image

DevEval: A Manually-Annotated Code Generation Benchmark Aligned with Real-World Code Repositories

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Li, Jia, Li, Ge, Zhao, Yunfei, Li, Yongmin, Liu, Huanyu, Zhu, Hao, Wang, Lecheng, Liu, Kaibo, Fang, Zheng, Wang, Lanshen, Ding, Jiazheng, Zhang, Xuanming, Zhu, Yuqi, Dong, Yihong, Jin, Zhi, Li, Binhua, Huang, Fei, Li, Yongbin
Format:	Preprint
Published:	2024
Subjects:	Computation and Language Software Engineering
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

DevEval: Evaluating Code Generation in Practical Software Projects
by: Li, Jia, et al.
Published: (2024)

EvoCodeBench: An Evolving Code Generation Benchmark Aligned with Real-World Code Repositories
by: Li, Jia, et al.
Published: (2024)

EvoCodeBench: An Evolving Code Generation Benchmark with Domain-Specific Evaluations
by: Li, Jia, et al.
Published: (2024)

Showing LLM-Generated Code Selectively Based on Confidence of LLMs
by: Li, Jia, et al.
Published: (2024)

From I/O to Code with Discovery Agent
by: Dong, Yihong, et al.
Published: (2026)

Think Anywhere in Code Generation
by: Jiang, Xue, et al.
Published: (2026)

CodeScore: Evaluating Code Generation by Learning Code Execution
by: Dong, Yihong, et al.
Published: (2023)

CodeRL+: Improving Code Generation via Reinforcement with Execution Semantics Alignment
by: Jiang, Xue, et al.
Published: (2025)

aiXcoder-7B-v2: Training LLMs to Fully Utilize the Long Context in Repository-level Code Completion
by: Li, Jia, et al.
Published: (2025)

Generating Equivalent Representations of Code By A Self-Reflection Approach
by: Li, Jia, et al.
Published: (2024)

Do Code LLMs Understand Design Patterns?
by: Pan, Zhenyu, et al.
Published: (2025)

Large Language Model Unlearning for Source Code
by: Jiang, Xue, et al.
Published: (2025)

Uncertainty-Guided Chain-of-Thought for Code Generation with LLMs
by: Zhu, Yuqi, et al.
Published: (2025)

CodeMEM: AST-Guided Adaptive Memory for Repository-Level Iterative Code Generation
by: Wang, Peiding, et al.
Published: (2026)

aiXcoder-7B: A Lightweight and Effective Large Language Model for Code Processing
by: Jiang, Siyuan, et al.
Published: (2024)

CodeDPO: Aligning Code Models with Self Generated and Verified Source Code
by: Zhang, Kechi, et al.
Published: (2024)

Alibaba LingmaAgent: Improving Automated Issue Resolution via Comprehensive Repository Exploration
by: Ma, Yingwei, et al.
Published: (2024)

VulInstruct: Teaching LLMs Root-Cause Reasoning for Vulnerability Detection via Security Specifications
by: Zhu, Hao, et al.
Published: (2025)

Theoretical Proof that Auto-regressive Language Models Collapse when Real-world Data is a Finite Set
by: Wang, Lecheng, et al.
Published: (2024)

Codev-Bench: How Do LLMs Understand Developer-Centric Code Completion?
by: Pan, Zhenyu, et al.
Published: (2024)

LLMs as Continuous Learners: Improving the Reproduction of Defective Code in Software Issues
by: Lin, Yalan, et al.
Published: (2024)

M2rc-Eval: Massively Multilingual Repository-level Code Completion Evaluation
by: Liu, Jiaheng, et al.
Published: (2024)

Self-planning Code Generation with Large Language Models
by: Jiang, Xue, et al.
Published: (2023)

SATURN: SAT-based Reinforcement Learning to Unleash LLMs Reasoning
by: Liu, Huanyu, et al.
Published: (2025)

ROCODE: Integrating Backtracking Mechanism and Program Analysis in Large Language Models for Code Generation
by: Jiang, Xue, et al.
Published: (2024)

To Diff or Not to Diff? Structure-Aware and Adaptive Output Formats for Efficient LLM-based Code Editing
by: Cheng, Wei, et al.
Published: (2026)

EvoCoT: Overcoming the Exploration Bottleneck in Reinforcement Learning
by: Liu, Huanyu, et al.
Published: (2025)

GraphCodeAgent: Dual Graph-Guided LLM Agent for Retrieval-Augmented Repo-Level Code Generation
by: Li, Jia, et al.
Published: (2025)

Saber: An Efficient Sampling with Adaptive Acceleration and Backtracking Enhanced Remasking for Diffusion Language Model
by: Dong, Yihong, et al.
Published: (2025)

MdEval: Massively Multilingual Code Debugging
by: Liu, Shukai, et al.
Published: (2024)

McEval: Massively Multilingual Code Evaluation
by: Chai, Linzheng, et al.
Published: (2024)

Python Symbolic Execution with LLM-powered Code Generation
by: Wang, Wenhan, et al.
Published: (2024)

Rethinking Repetition Problems of LLMs in Code Generation
by: Dong, Yihong, et al.
Published: (2025)

Teaching Code LLMs to Use Autocompletion Tools in Repository-Level Code Generation
by: Wang, Chong, et al.
Published: (2024)

Understanding and Mitigating Errors of LLM-Generated RTL Code
by: Zhang, Jiazheng, et al.
Published: (2025)

RepoMirage: Probing Repository Context Reasoning in Code Agents with Perturbations
by: Li, Hanyu, et al.
Published: (2026)

Line-level Semantic Structure Learning for Code Vulnerability Detection
by: Wang, Ziliang, et al.
Published: (2024)

RL-PLUS: Countering Capability Boundary Collapse of LLMs in Reinforcement Learning with Hybrid-policy Optimization
by: Dong, Yihong, et al.
Published: (2025)

RepoSummary: Feature-Oriented Summarization and Documentation Generation for Code Repositories
by: Zhu, Yifeng, et al.
Published: (2025)

Self-collaboration Code Generation via ChatGPT
by: Dong, Yihong, et al.
Published: (2023)