Saved in:
| Main Authors: | , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2504.08703 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908335668723712 |
|---|---|
| author | Rashid, Muhammad Shihab Bock, Christian Zhuang, Yuan Buchholz, Alexander Esler, Tim Valentin, Simon Franceschi, Luca Wistuba, Martin Sivaprasad, Prabhu Teja Kim, Woo Jung Deoras, Anoop Zappella, Giovanni Callot, Laurent |
| author_facet | Rashid, Muhammad Shihab Bock, Christian Zhuang, Yuan Buchholz, Alexander Esler, Tim Valentin, Simon Franceschi, Luca Wistuba, Martin Sivaprasad, Prabhu Teja Kim, Woo Jung Deoras, Anoop Zappella, Giovanni Callot, Laurent |
| contents | Coding agents powered by large language models have shown impressive capabilities in software engineering tasks, but evaluating their performance across diverse programming languages and real-world scenarios remains challenging. We introduce SWE-PolyBench, a new multi-language benchmark for repository-level, execution-based evaluation of coding agents. SWE-PolyBench contains 2110 instances from 21 repositories and includes tasks in Java (165), JavaScript (1017), TypeScript (729) and Python (199), covering bug fixes, feature additions, and code refactoring. We provide a task and repository-stratified subsample (SWE-PolyBench500) and release an evaluation harness allowing for fully automated evaluation. To enable a more comprehensive comparison of coding agents, this work also presents a novel set of metrics rooted in syntax tree analysis. We evaluate leading open source coding agents on SWE-PolyBench, revealing their strengths and limitations across languages, task types, and complexity classes. Our experiments show that current agents exhibit uneven performances across languages and struggle with complex problems while showing higher performance on simpler tasks. SWE-PolyBench aims to drive progress in developing more versatile and robust AI coding assistants for real-world software engineering. Our datasets and code are available at: https://github.com/amazon-science/SWE-PolyBench |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_08703 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | SWE-PolyBench: A multi-language benchmark for repository level evaluation of coding agents Rashid, Muhammad Shihab Bock, Christian Zhuang, Yuan Buchholz, Alexander Esler, Tim Valentin, Simon Franceschi, Luca Wistuba, Martin Sivaprasad, Prabhu Teja Kim, Woo Jung Deoras, Anoop Zappella, Giovanni Callot, Laurent Software Engineering Coding agents powered by large language models have shown impressive capabilities in software engineering tasks, but evaluating their performance across diverse programming languages and real-world scenarios remains challenging. We introduce SWE-PolyBench, a new multi-language benchmark for repository-level, execution-based evaluation of coding agents. SWE-PolyBench contains 2110 instances from 21 repositories and includes tasks in Java (165), JavaScript (1017), TypeScript (729) and Python (199), covering bug fixes, feature additions, and code refactoring. We provide a task and repository-stratified subsample (SWE-PolyBench500) and release an evaluation harness allowing for fully automated evaluation. To enable a more comprehensive comparison of coding agents, this work also presents a novel set of metrics rooted in syntax tree analysis. We evaluate leading open source coding agents on SWE-PolyBench, revealing their strengths and limitations across languages, task types, and complexity classes. Our experiments show that current agents exhibit uneven performances across languages and struggle with complex problems while showing higher performance on simpler tasks. SWE-PolyBench aims to drive progress in developing more versatile and robust AI coding assistants for real-world software engineering. Our datasets and code are available at: https://github.com/amazon-science/SWE-PolyBench |
| title | SWE-PolyBench: A multi-language benchmark for repository level evaluation of coding agents |
| topic | Software Engineering |
| url | https://arxiv.org/abs/2504.08703 |