Saved in:
| Main Authors: | , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2605.15846 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918510002700288 |
|---|---|
| author | Xu, Xinbo Yang, Ruihan Shen, Haiyang Xu, Wendong Gao, Bofei Wu, Ruoyu Shi, Kean Xie, Weichu Chen, Xuanzhong Wu, Ming Zeng, Jason Heinrich, Michael Zhang, Elvis Chen, Liang Li, Kuan Chang, Baobao |
| author_facet | Xu, Xinbo Yang, Ruihan Shen, Haiyang Xu, Wendong Gao, Bofei Wu, Ruoyu Shi, Kean Xie, Weichu Chen, Xuanzhong Wu, Ming Zeng, Jason Heinrich, Michael Zhang, Elvis Chen, Liang Li, Kuan Chang, Baobao |
| contents | Coding agents are increasingly deployed in real software development, where a single version iteration requires months of coordinated work across many files. However, most existing benchmarks focus predominantly on single-issue bug fixes from Python repositories, with coarse pass/fail evaluation outcomes, and thus fail to capture long-horizon, multi-target development at real engineering scale. To address this gap, we present RoadmapBench, a benchmark of 115 long-horizon coding tasks grounded in real open-source version upgrades across 17 repositories and 5 programming languages. Each task places the agent on a source-version code snapshot and provides a multi-target roadmap instruction requiring it to implement the functionality introduced in the target version, with a median modification of 3,700 lines across 51 files. We conduct a systematic evaluation on thirteen frontier models and find that even the strongest, Claude-Opus-4.7, resolves only 39.1% of tasks, while the weakest achieves merely 5.2%, in stark contrast to existing bug-fix benchmarks, suggesting that long-horizon software development remains a largely unsolved problem. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_15846 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | RoadmapBench: Evaluating Long-Horizon Agentic Software Development Across Version Upgrades Xu, Xinbo Yang, Ruihan Shen, Haiyang Xu, Wendong Gao, Bofei Wu, Ruoyu Shi, Kean Xie, Weichu Chen, Xuanzhong Wu, Ming Zeng, Jason Heinrich, Michael Zhang, Elvis Chen, Liang Li, Kuan Chang, Baobao Software Engineering Artificial Intelligence Coding agents are increasingly deployed in real software development, where a single version iteration requires months of coordinated work across many files. However, most existing benchmarks focus predominantly on single-issue bug fixes from Python repositories, with coarse pass/fail evaluation outcomes, and thus fail to capture long-horizon, multi-target development at real engineering scale. To address this gap, we present RoadmapBench, a benchmark of 115 long-horizon coding tasks grounded in real open-source version upgrades across 17 repositories and 5 programming languages. Each task places the agent on a source-version code snapshot and provides a multi-target roadmap instruction requiring it to implement the functionality introduced in the target version, with a median modification of 3,700 lines across 51 files. We conduct a systematic evaluation on thirteen frontier models and find that even the strongest, Claude-Opus-4.7, resolves only 39.1% of tasks, while the weakest achieves merely 5.2%, in stark contrast to existing bug-fix benchmarks, suggesting that long-horizon software development remains a largely unsolved problem. |
| title | RoadmapBench: Evaluating Long-Horizon Agentic Software Development Across Version Upgrades |
| topic | Software Engineering Artificial Intelligence |
| url | https://arxiv.org/abs/2605.15846 |