SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915619040919552 |
|---|---|
| author | Deng, Xiang Da, Jeff Pan, Edwin He, Yannis Yiming Ide, Charles Garg, Kanak Lauffer, Niklas Park, Andrew Pasari, Nitin Rane, Chetan Sampath, Karmini Krishnan, Maya Kundurthy, Srivatsa Hendryx, Sean Wang, Zifan Bharadwaj, Vijay Holm, Jeff Aluri, Raja Zhang, Chen Bo Calvin Jacobson, Noah Liu, Bing Kenstler, Brad |
| author_facet | Deng, Xiang Da, Jeff Pan, Edwin He, Yannis Yiming Ide, Charles Garg, Kanak Lauffer, Niklas Park, Andrew Pasari, Nitin Rane, Chetan Sampath, Karmini Krishnan, Maya Kundurthy, Srivatsa Hendryx, Sean Wang, Zifan Bharadwaj, Vijay Holm, Jeff Aluri, Raja Zhang, Chen Bo Calvin Jacobson, Noah Liu, Bing Kenstler, Brad |
| contents | We introduce SWE-Bench Pro, a substantially more challenging benchmark that builds upon the best practices of SWE-BENCH [25], but is explicitly designed to capture realistic, complex, enterprise-level problems beyond the scope of SWE-BENCH. SWE-BENCH PRO contains 1,865 problems sourced from a diverse set of 41 actively maintained repositories spanning business applications, B2B services, and developer tools. The benchmark is partitioned into a public set with open access to problems sourced from 11 repositories, a held-out set of 12 repositories and a commercial set of 18 proprietary repositories where we have formal partnership agreements with early-stage startups. Problems in the held-out and the commercial set are not publicly accessible, but we release results on the commercial set. Our benchmark features long-horizon tasks that may require hours to days for a professional software engineer to complete, often involving patches across multiple files and substantial code modifications. All tasks are human-verified and augmented with sufficient context to ensure resolvability. To better understand these limitations, we cluster the failure modes observed in the collected agent trajectories for a clearer characterization of the error patterns exhibited by current models. Overall, SWE-BENCH PRO provides a contamination-resistant testbed that more faithfully captures the complexity and diversity of real-world software development, advancing the pursuit of truly autonomous software engineering agents at a professional level. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_16941 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks? Deng, Xiang Da, Jeff Pan, Edwin He, Yannis Yiming Ide, Charles Garg, Kanak Lauffer, Niklas Park, Andrew Pasari, Nitin Rane, Chetan Sampath, Karmini Krishnan, Maya Kundurthy, Srivatsa Hendryx, Sean Wang, Zifan Bharadwaj, Vijay Holm, Jeff Aluri, Raja Zhang, Chen Bo Calvin Jacobson, Noah Liu, Bing Kenstler, Brad Software Engineering Computation and Language We introduce SWE-Bench Pro, a substantially more challenging benchmark that builds upon the best practices of SWE-BENCH [25], but is explicitly designed to capture realistic, complex, enterprise-level problems beyond the scope of SWE-BENCH. SWE-BENCH PRO contains 1,865 problems sourced from a diverse set of 41 actively maintained repositories spanning business applications, B2B services, and developer tools. The benchmark is partitioned into a public set with open access to problems sourced from 11 repositories, a held-out set of 12 repositories and a commercial set of 18 proprietary repositories where we have formal partnership agreements with early-stage startups. Problems in the held-out and the commercial set are not publicly accessible, but we release results on the commercial set. Our benchmark features long-horizon tasks that may require hours to days for a professional software engineer to complete, often involving patches across multiple files and substantial code modifications. All tasks are human-verified and augmented with sufficient context to ensure resolvability. To better understand these limitations, we cluster the failure modes observed in the collected agent trajectories for a clearer characterization of the error patterns exhibited by current models. Overall, SWE-BENCH PRO provides a contamination-resistant testbed that more faithfully captures the complexity and diversity of real-world software development, advancing the pursuit of truly autonomous software engineering agents at a professional level. |
| title | SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks? |
| topic | Software Engineering Computation and Language |
| url | https://arxiv.org/abs/2509.16941 |