Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2512.12730 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912809272475648 |
|---|---|
| author | Ding, Jingzhe Long, Shengda Pu, Changxin Zhou, Huan Gao, Hongwan Gao, Xiang He, Chao Hou, Yue Hu, Fei Li, Zhaojian Shi, Weiran Wang, Zaiyuan Zan, Daoguang Zhang, Chenchen Zhang, Xiaoxu Chen, Qizhi Cheng, Xianfu Deng, Bo Gu, Qingshui Hua, Kai Lin, Juntao Liu, Pai Li, Mingchen Pan, Xuanguang Peng, Zifan Qin, Yujia Shan, Yong Tan, Zhewen Xie, Weihao Wang, Zihan Yuan, Yishuo Zhang, Jiayu Zhao, Enduo Zhao, Yunfei Zhu, He Zhu, Liya Zou, Chenyang Ding, Ming Jiao, Jianpeng Liu, Jiaheng Liu, Minghao Liu, Qian Tao, Chongyang Yang, Jian Yang, Tong Zhang, Zhaoxiang Chen, Xinjie Huang, Wenhao Zhang, Ge |
| author_facet | Ding, Jingzhe Long, Shengda Pu, Changxin Zhou, Huan Gao, Hongwan Gao, Xiang He, Chao Hou, Yue Hu, Fei Li, Zhaojian Shi, Weiran Wang, Zaiyuan Zan, Daoguang Zhang, Chenchen Zhang, Xiaoxu Chen, Qizhi Cheng, Xianfu Deng, Bo Gu, Qingshui Hua, Kai Lin, Juntao Liu, Pai Li, Mingchen Pan, Xuanguang Peng, Zifan Qin, Yujia Shan, Yong Tan, Zhewen Xie, Weihao Wang, Zihan Yuan, Yishuo Zhang, Jiayu Zhao, Enduo Zhao, Yunfei Zhu, He Zhu, Liya Zou, Chenyang Ding, Ming Jiao, Jianpeng Liu, Jiaheng Liu, Minghao Liu, Qian Tao, Chongyang Yang, Jian Yang, Tong Zhang, Zhaoxiang Chen, Xinjie Huang, Wenhao Zhang, Ge |
| contents | Recent advances in coding agents suggest rapid progress toward autonomous software development, yet existing benchmarks fail to rigorously evaluate the long-horizon capabilities required to build complete software systems. Most prior evaluations focus on localized code generation, scaffolded completion, or short-term repair tasks, leaving open the question of whether agents can sustain coherent reasoning, planning, and execution over the extended horizons demanded by real-world repository construction. To address this gap, we present NL2Repo Bench, a benchmark explicitly designed to evaluate the long-horizon repository generation ability of coding agents. Given only a single natural-language requirements document and an empty workspace, agents must autonomously design the architecture, manage dependencies, implement multi-module logic, and produce a fully installable Python library. Our experiments across state-of-the-art open- and closed-source models reveal that long-horizon repository generation remains largely unsolved: even the strongest agents achieve below 40% average test pass rates and rarely complete an entire repository correctly. Detailed analysis uncovers fundamental long-horizon failure modes, including premature termination, loss of global coherence, fragile cross-file dependencies, and inadequate planning over hundreds of interaction steps. NL2Repo Bench establishes a rigorous, verifiable testbed for measuring sustained agentic competence and highlights long-horizon reasoning as a central bottleneck for the next generation of autonomous coding agents. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_12730 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | NL2Repo-Bench: Towards Long-Horizon Repository Generation Evaluation of Coding Agents Ding, Jingzhe Long, Shengda Pu, Changxin Zhou, Huan Gao, Hongwan Gao, Xiang He, Chao Hou, Yue Hu, Fei Li, Zhaojian Shi, Weiran Wang, Zaiyuan Zan, Daoguang Zhang, Chenchen Zhang, Xiaoxu Chen, Qizhi Cheng, Xianfu Deng, Bo Gu, Qingshui Hua, Kai Lin, Juntao Liu, Pai Li, Mingchen Pan, Xuanguang Peng, Zifan Qin, Yujia Shan, Yong Tan, Zhewen Xie, Weihao Wang, Zihan Yuan, Yishuo Zhang, Jiayu Zhao, Enduo Zhao, Yunfei Zhu, He Zhu, Liya Zou, Chenyang Ding, Ming Jiao, Jianpeng Liu, Jiaheng Liu, Minghao Liu, Qian Tao, Chongyang Yang, Jian Yang, Tong Zhang, Zhaoxiang Chen, Xinjie Huang, Wenhao Zhang, Ge Computation and Language Recent advances in coding agents suggest rapid progress toward autonomous software development, yet existing benchmarks fail to rigorously evaluate the long-horizon capabilities required to build complete software systems. Most prior evaluations focus on localized code generation, scaffolded completion, or short-term repair tasks, leaving open the question of whether agents can sustain coherent reasoning, planning, and execution over the extended horizons demanded by real-world repository construction. To address this gap, we present NL2Repo Bench, a benchmark explicitly designed to evaluate the long-horizon repository generation ability of coding agents. Given only a single natural-language requirements document and an empty workspace, agents must autonomously design the architecture, manage dependencies, implement multi-module logic, and produce a fully installable Python library. Our experiments across state-of-the-art open- and closed-source models reveal that long-horizon repository generation remains largely unsolved: even the strongest agents achieve below 40% average test pass rates and rarely complete an entire repository correctly. Detailed analysis uncovers fundamental long-horizon failure modes, including premature termination, loss of global coherence, fragile cross-file dependencies, and inadequate planning over hundreds of interaction steps. NL2Repo Bench establishes a rigorous, verifiable testbed for measuring sustained agentic competence and highlights long-horizon reasoning as a central bottleneck for the next generation of autonomous coding agents. |
| title | NL2Repo-Bench: Towards Long-Horizon Repository Generation Evaluation of Coding Agents |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2512.12730 |