Benchmark Test-Time Scaling of General LLM Agents
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2026
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866918349680672768 |
|---|---|
| author | Li, Xiaochuan Ming, Ryan Setlur, Pranav Paladugu, Abhijay Tang, Andy Kang, Hao Shao, Shuai Jin, Rong Xiong, Chenyan |
| author_facet | Li, Xiaochuan Ming, Ryan Setlur, Pranav Paladugu, Abhijay Tang, Andy Kang, Hao Shao, Shuai Jin, Rong Xiong, Chenyan |
| contents | LLM agents are increasingly expected to function as general-purpose systems capable of resolving open-ended user requests. While existing benchmarks focus on domain-aware environments for developing specialized agents, evaluating general-purpose agents requires more realistic settings that challenge them to operate across multiple skills and tools within a unified environment. We introduce General AgentBench, a benchmark that provides such a unified framework for evaluating general LLM agents across search, coding, reasoning, and tool-use domains. Using General AgentBench, we systematically study test-time scaling behaviors under sequential scaling (iterative interaction) and parallel scaling (sampling multiple trajectories). Evaluation of ten leading LLM agents reveals a substantial performance degradation when moving from domain-specific evaluations to this general-agent setting. Moreover, we find that neither scaling methodology yields effective performance improvements in practice, due to two fundamental limitations: context ceiling in sequential scaling and verification gap in parallel scaling. Code is publicly available at https://github.com/cxcscmu/General-AgentBench. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_18998 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Benchmark Test-Time Scaling of General LLM Agents Li, Xiaochuan Ming, Ryan Setlur, Pranav Paladugu, Abhijay Tang, Andy Kang, Hao Shao, Shuai Jin, Rong Xiong, Chenyan Artificial Intelligence Computation and Language LLM agents are increasingly expected to function as general-purpose systems capable of resolving open-ended user requests. While existing benchmarks focus on domain-aware environments for developing specialized agents, evaluating general-purpose agents requires more realistic settings that challenge them to operate across multiple skills and tools within a unified environment. We introduce General AgentBench, a benchmark that provides such a unified framework for evaluating general LLM agents across search, coding, reasoning, and tool-use domains. Using General AgentBench, we systematically study test-time scaling behaviors under sequential scaling (iterative interaction) and parallel scaling (sampling multiple trajectories). Evaluation of ten leading LLM agents reveals a substantial performance degradation when moving from domain-specific evaluations to this general-agent setting. Moreover, we find that neither scaling methodology yields effective performance improvements in practice, due to two fundamental limitations: context ceiling in sequential scaling and verification gap in parallel scaling. Code is publicly available at https://github.com/cxcscmu/General-AgentBench. |
| title | Benchmark Test-Time Scaling of General LLM Agents |
| topic | Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2602.18998 |