EDIT-Bench: Evaluating LLM Abilities to Perform Real-World Instructed Code Edits
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912714801020928 |
|---|---|
| author | Chi, Wayne Chen, Valerie Shar, Ryan Mittal, Aditya Liang, Jenny Chiang, Wei-Lin Angelopoulos, Anastasios Nikolas Stoica, Ion Neubig, Graham Talwalkar, Ameet Donahue, Chris |
| author_facet | Chi, Wayne Chen, Valerie Shar, Ryan Mittal, Aditya Liang, Jenny Chiang, Wei-Lin Angelopoulos, Anastasios Nikolas Stoica, Ion Neubig, Graham Talwalkar, Ameet Donahue, Chris |
| contents | Instructed code editing, where LLMs directly modify a developer's existing code based on a user instruction, is becoming a widely used interaction mode in AI coding assistants. However, few benchmarks directly evaluate this capability and current datasets often rely on artificial sources. We introduce EDIT-Bench, a benchmark for evaluating LLM code editing capabilities grounded in real-world usage, i.e., user instructions and code contexts collected in the wild. EDIT-Bench comprises of 540 problems, multiple natural and programming languages, and a diverse set of real-world use cases, ranging from resolving errors to adding features. EDIT-Bench introduces context-dependent problems that require the model to understand code context, highlighted code, and cursor position in addition to the user instruction. We evaluate 40 diverse LLMs and observe that EDIT-Bench is a challenging set of problems where only 1 model scores over 60%. We find that model performance varies across different categories of user instructions. Further, we find that varying levels of contextual information greatly affect task success rate, with performance varying up to 11%, indicating the importance of evaluating with realistic context. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_04486 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | EDIT-Bench: Evaluating LLM Abilities to Perform Real-World Instructed Code Edits Chi, Wayne Chen, Valerie Shar, Ryan Mittal, Aditya Liang, Jenny Chiang, Wei-Lin Angelopoulos, Anastasios Nikolas Stoica, Ion Neubig, Graham Talwalkar, Ameet Donahue, Chris Software Engineering Instructed code editing, where LLMs directly modify a developer's existing code based on a user instruction, is becoming a widely used interaction mode in AI coding assistants. However, few benchmarks directly evaluate this capability and current datasets often rely on artificial sources. We introduce EDIT-Bench, a benchmark for evaluating LLM code editing capabilities grounded in real-world usage, i.e., user instructions and code contexts collected in the wild. EDIT-Bench comprises of 540 problems, multiple natural and programming languages, and a diverse set of real-world use cases, ranging from resolving errors to adding features. EDIT-Bench introduces context-dependent problems that require the model to understand code context, highlighted code, and cursor position in addition to the user instruction. We evaluate 40 diverse LLMs and observe that EDIT-Bench is a challenging set of problems where only 1 model scores over 60%. We find that model performance varies across different categories of user instructions. Further, we find that varying levels of contextual information greatly affect task success rate, with performance varying up to 11%, indicating the importance of evaluating with realistic context. |
| title | EDIT-Bench: Evaluating LLM Abilities to Perform Real-World Instructed Code Edits |
| topic | Software Engineering |
| url | https://arxiv.org/abs/2511.04486 |