LongCLI-Bench: A Preliminary Benchmark and Study for Long-horizon Agentic Programming in Command-Line Interfaces
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2026
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866917294741913600 |
|---|---|
| author | Feng, Yukang Sun, Jianwen Yang, Zelai Ai, Jiaxin Li, Chuanhao Li, Zizhen Zhang, Fanrui He, Kang Ma, Rui Lin, Jifan Sun, Jie Xiao, Yang Zhou, Sizhuo Wu, Wenxiao Liu, Yiming Liu, Pengfei Qiao, Yu Zhang, Shenglin Zhang, Kaipeng |
| author_facet | Feng, Yukang Sun, Jianwen Yang, Zelai Ai, Jiaxin Li, Chuanhao Li, Zizhen Zhang, Fanrui He, Kang Ma, Rui Lin, Jifan Sun, Jie Xiao, Yang Zhou, Sizhuo Wu, Wenxiao Liu, Yiming Liu, Pengfei Qiao, Yu Zhang, Shenglin Zhang, Kaipeng |
| contents | Recent advances in AI-assisted programming have empowered agents to execute complex workflows via command-line interfaces, however, existing benchmarks are limited by short task horizons, data contamination from GitHub scraping, and a lack of fine-grained evaluation metrics, fail to rigorously evaluate the long-horizon planning and execution capabilities essential for realistic software engineering. To address these gaps, we introduce LongCLI-Bench, a comprehensive benchmark designed to evaluate agentic capabilities across long-horizon, realistic tasks. We curated 20 high-quality, long-horizon tasks from over 1,000 computer science assignments and real-world workflows, covering four engineering categories: from scratch, feature addition, bug fixing, and refactoring. We propose a dual-set testing protocol for LongCLI-Bench, which measures requirement fulfillment (fail-to-pass) and regression avoidance (pass-to-pass), and incorporates step-level scoring to pinpoint execution failures. Extensive experiments reveal that even state-of-the-art agents achieve pass rates below 20% in LongCLI-Bench. Step-level analysis further indicates that the majority of tasks stall at less than 30% completion, highlighting that critical failures often occur in the early stages. Although self-correction offers marginal gains, human-agent collaboration through plan injection and interactive guidance yields significantly higher improvements. These results highlight that future research must emphasize the development of synergistic human-agent workflows alongside advances in agents' planning and execution capabilities to overcome key challenges in long-horizon task performance. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_14337 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | LongCLI-Bench: A Preliminary Benchmark and Study for Long-horizon Agentic Programming in Command-Line Interfaces Feng, Yukang Sun, Jianwen Yang, Zelai Ai, Jiaxin Li, Chuanhao Li, Zizhen Zhang, Fanrui He, Kang Ma, Rui Lin, Jifan Sun, Jie Xiao, Yang Zhou, Sizhuo Wu, Wenxiao Liu, Yiming Liu, Pengfei Qiao, Yu Zhang, Shenglin Zhang, Kaipeng Software Engineering Multiagent Systems Recent advances in AI-assisted programming have empowered agents to execute complex workflows via command-line interfaces, however, existing benchmarks are limited by short task horizons, data contamination from GitHub scraping, and a lack of fine-grained evaluation metrics, fail to rigorously evaluate the long-horizon planning and execution capabilities essential for realistic software engineering. To address these gaps, we introduce LongCLI-Bench, a comprehensive benchmark designed to evaluate agentic capabilities across long-horizon, realistic tasks. We curated 20 high-quality, long-horizon tasks from over 1,000 computer science assignments and real-world workflows, covering four engineering categories: from scratch, feature addition, bug fixing, and refactoring. We propose a dual-set testing protocol for LongCLI-Bench, which measures requirement fulfillment (fail-to-pass) and regression avoidance (pass-to-pass), and incorporates step-level scoring to pinpoint execution failures. Extensive experiments reveal that even state-of-the-art agents achieve pass rates below 20% in LongCLI-Bench. Step-level analysis further indicates that the majority of tasks stall at less than 30% completion, highlighting that critical failures often occur in the early stages. Although self-correction offers marginal gains, human-agent collaboration through plan injection and interactive guidance yields significantly higher improvements. These results highlight that future research must emphasize the development of synergistic human-agent workflows alongside advances in agents' planning and execution capabilities to overcome key challenges in long-horizon task performance. |
| title | LongCLI-Bench: A Preliminary Benchmark and Study for Long-horizon Agentic Programming in Command-Line Interfaces |
| topic | Software Engineering Multiagent Systems |
| url | https://arxiv.org/abs/2602.14337 |