IDE-Bench: Evaluating Large Language Models as IDE Agents on Real-World Software Engineering Tasks
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2026
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866917233172676608 |
|---|---|
| author | Mateega, Spencer Yang, Jeff Costello, Tiana Jadhav, Shaurya Tian, Nicole Garcinuño, Agustin |
| author_facet | Mateega, Spencer Yang, Jeff Costello, Tiana Jadhav, Shaurya Tian, Nicole Garcinuño, Agustin |
| contents | IDE-Bench is a comprehensive framework for evaluating AI IDE agents on real-world software engineering tasks through an IDE-native tool interface. We present a Dockerized test harness that goes beyond raw terminal execution, granting models a structured tool ecosystem that represents AI-native IDEs like Cursor and Windsurf. By providing high-level abstractions for codebase search, structured file editing, and tools for testing full-stack applications, IDE-Bench evaluates an agent's ability to act as a true engineering collaborator. For evaluation and to prevent training data contamination, we created 80 tasks across eight never-published repositories spanning C/C++, Java, and MERN stacks, representing modern tech stack production scenarios, including feature implementation, bug fixing, refactoring, and performance optimization that mirror daily developer workflows in private codebases. Our benchmark is the first to systematically correlate agent-reported intent with successful project-level modifications in a multi-language, full-stack environment on completely uncontaminated code. We release IDE-Bench and a public leaderboard at: https://ide-bench.com. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_20886 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | IDE-Bench: Evaluating Large Language Models as IDE Agents on Real-World Software Engineering Tasks Mateega, Spencer Yang, Jeff Costello, Tiana Jadhav, Shaurya Tian, Nicole Garcinuño, Agustin Software Engineering Machine Learning IDE-Bench is a comprehensive framework for evaluating AI IDE agents on real-world software engineering tasks through an IDE-native tool interface. We present a Dockerized test harness that goes beyond raw terminal execution, granting models a structured tool ecosystem that represents AI-native IDEs like Cursor and Windsurf. By providing high-level abstractions for codebase search, structured file editing, and tools for testing full-stack applications, IDE-Bench evaluates an agent's ability to act as a true engineering collaborator. For evaluation and to prevent training data contamination, we created 80 tasks across eight never-published repositories spanning C/C++, Java, and MERN stacks, representing modern tech stack production scenarios, including feature implementation, bug fixing, refactoring, and performance optimization that mirror daily developer workflows in private codebases. Our benchmark is the first to systematically correlate agent-reported intent with successful project-level modifications in a multi-language, full-stack environment on completely uncontaminated code. We release IDE-Bench and a public leaderboard at: https://ide-bench.com. |
| title | IDE-Bench: Evaluating Large Language Models as IDE Agents on Real-World Software Engineering Tasks |
| topic | Software Engineering Machine Learning |
| url | https://arxiv.org/abs/2601.20886 |