OctoBench: Benchmarking Scaffold-Aware Instruction Following in Repository-Grounded Agentic Coding
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911378634178560 |
|---|---|
| author | Ding, Deming Liu, Shichun Yang, Enhui Lin, Jiahang Chen, Ziying Dou, Shihan Guo, Honglin Cheng, Weiyu Zhao, Pengyu Xiao, Chengjun Zeng, Qunhong Zhang, Qi Huang, Xuanjing Xu, Qidi Gui, Tao |
| author_facet | Ding, Deming Liu, Shichun Yang, Enhui Lin, Jiahang Chen, Ziying Dou, Shihan Guo, Honglin Cheng, Weiyu Zhao, Pengyu Xiao, Chengjun Zeng, Qunhong Zhang, Qi Huang, Xuanjing Xu, Qidi Gui, Tao |
| contents | Modern coding scaffolds turn LLMs into capable software agents, but their ability to follow scaffold-specified instructions remains under-examined, especially when constraints are heterogeneous and persist across interactions. To fill this gap, we introduce OctoBench, which benchmarks scaffold-aware instruction following in repository-grounded agentic coding. OctoBench includes 34 environments and 217 tasks instantiated under three scaffold types, and is paired with 7,098 objective checklist items. To disentangle solving the task from following the rules, we provide an automated observation-and-scoring toolkit that captures full trajectories and performs fine-grained checks. Experiments on eight representative models reveal a systematic gap between task-solving and scaffold-aware compliance, underscoring the need for training and evaluation that explicitly targets heterogeneous instruction following. We release the benchmark to support reproducible benchmarking and to accelerate the development of more scaffold-aware coding agents. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_10343 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | OctoBench: Benchmarking Scaffold-Aware Instruction Following in Repository-Grounded Agentic Coding Ding, Deming Liu, Shichun Yang, Enhui Lin, Jiahang Chen, Ziying Dou, Shihan Guo, Honglin Cheng, Weiyu Zhao, Pengyu Xiao, Chengjun Zeng, Qunhong Zhang, Qi Huang, Xuanjing Xu, Qidi Gui, Tao Computation and Language Artificial Intelligence Modern coding scaffolds turn LLMs into capable software agents, but their ability to follow scaffold-specified instructions remains under-examined, especially when constraints are heterogeneous and persist across interactions. To fill this gap, we introduce OctoBench, which benchmarks scaffold-aware instruction following in repository-grounded agentic coding. OctoBench includes 34 environments and 217 tasks instantiated under three scaffold types, and is paired with 7,098 objective checklist items. To disentangle solving the task from following the rules, we provide an automated observation-and-scoring toolkit that captures full trajectories and performs fine-grained checks. Experiments on eight representative models reveal a systematic gap between task-solving and scaffold-aware compliance, underscoring the need for training and evaluation that explicitly targets heterogeneous instruction following. We release the benchmark to support reproducible benchmarking and to accelerate the development of more scaffold-aware coding agents. |
| title | OctoBench: Benchmarking Scaffold-Aware Instruction Following in Repository-Grounded Agentic Coding |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2601.10343 |