AgentLongBench: A Controllable Long Benchmark For Long-Contexts Agents via Environment Rollouts
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2026
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866917234374344704 |
|---|---|
| author | Fang, Shicheng Wang, Yuxin Liu, Xiaoran Lu, Jiahao Tan, Chuanyuan Chen, Xinchi Zheng, Yining Huang, Xuanjing Qiu, Xipeng |
| author_facet | Fang, Shicheng Wang, Yuxin Liu, Xiaoran Lu, Jiahao Tan, Chuanyuan Chen, Xinchi Zheng, Yining Huang, Xuanjing Qiu, Xipeng |
| contents | The evolution of Large Language Models (LLMs) into autonomous agents necessitates the management of extensive, dynamic contexts. Current benchmarks, however, remain largely static, relying on passive retrieval tasks that fail to simulate the complexities of agent-environment interaction, such as non-linear reasoning and iterative feedback. To address this, we introduce \textbf{AgentLongBench}, which evaluates agents through simulated environment rollouts based on Lateral Thinking Puzzles. This framework generates rigorous interaction trajectories across knowledge-intensive and knowledge-free scenarios. Experiments with state-of-the-art models and memory systems (32K to 4M tokens) expose a critical weakness: while adept at static retrieval, agents struggle with the dynamic information synthesis essential for workflows. Our analysis indicates that this degradation is driven by the minimum number of tokens required to resolve a query. This factor explains why the high information density inherent in massive tool responses poses a significantly greater challenge than the memory fragmentation typical of long-turn dialogues. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_20730 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | AgentLongBench: A Controllable Long Benchmark For Long-Contexts Agents via Environment Rollouts Fang, Shicheng Wang, Yuxin Liu, Xiaoran Lu, Jiahao Tan, Chuanyuan Chen, Xinchi Zheng, Yining Huang, Xuanjing Qiu, Xipeng Computation and Language The evolution of Large Language Models (LLMs) into autonomous agents necessitates the management of extensive, dynamic contexts. Current benchmarks, however, remain largely static, relying on passive retrieval tasks that fail to simulate the complexities of agent-environment interaction, such as non-linear reasoning and iterative feedback. To address this, we introduce \textbf{AgentLongBench}, which evaluates agents through simulated environment rollouts based on Lateral Thinking Puzzles. This framework generates rigorous interaction trajectories across knowledge-intensive and knowledge-free scenarios. Experiments with state-of-the-art models and memory systems (32K to 4M tokens) expose a critical weakness: while adept at static retrieval, agents struggle with the dynamic information synthesis essential for workflows. Our analysis indicates that this degradation is driven by the minimum number of tokens required to resolve a query. This factor explains why the high information density inherent in massive tool responses poses a significantly greater challenge than the memory fragmentation typical of long-turn dialogues. |
| title | AgentLongBench: A Controllable Long Benchmark For Long-Contexts Agents via Environment Rollouts |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2601.20730 |