Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluation
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866908591277998080 |
|---|---|
| author | Kapoor, Sayash Stroebl, Benedikt Kirgis, Peter Nadgir, Nitya Siegel, Zachary S Wei, Boyi Xue, Tianci Chen, Ziru Chen, Felix Utpala, Saiteja Ndzomga, Franck Oruganty, Dheeraj Luskin, Sophie Liu, Kangheng Yu, Botao Arora, Amit Hahm, Dongyoon Trivedi, Harsh Sun, Huan Lee, Juyong Jin, Tengjun Mai, Yifan Zhou, Yifei Zhu, Yuxuan Bommasani, Rishi Kang, Daniel Song, Dawn Henderson, Peter Su, Yu Liang, Percy Narayanan, Arvind |
| author_facet | Kapoor, Sayash Stroebl, Benedikt Kirgis, Peter Nadgir, Nitya Siegel, Zachary S Wei, Boyi Xue, Tianci Chen, Ziru Chen, Felix Utpala, Saiteja Ndzomga, Franck Oruganty, Dheeraj Luskin, Sophie Liu, Kangheng Yu, Botao Arora, Amit Hahm, Dongyoon Trivedi, Harsh Sun, Huan Lee, Juyong Jin, Tengjun Mai, Yifan Zhou, Yifei Zhu, Yuxuan Bommasani, Rishi Kang, Daniel Song, Dawn Henderson, Peter Su, Yu Liang, Percy Narayanan, Arvind |
| contents | AI agents have been developed for complex real-world tasks from coding to customer service. But AI agent evaluations suffer from many challenges that undermine our understanding of how well agents really work. We introduce the Holistic Agent Leaderboard (HAL) to address these challenges. We make three main contributions. First, we provide a standardized evaluation harness that orchestrates parallel evaluations across hundreds of VMs, reducing evaluation time from weeks to hours while eliminating common implementation bugs. Second, we conduct three-dimensional analysis spanning models, scaffolds, and benchmarks. We validate the harness by conducting 21,730 agent rollouts across 9 models and 9 benchmarks in coding, web navigation, science, and customer service with a total cost of about $40,000. Our analysis reveals surprising insights, such as higher reasoning effort reducing accuracy in the majority of runs. Third, we use LLM-aided log inspection to uncover previously unreported behaviors, such as searching for the benchmark on HuggingFace instead of solving a task, or misusing credit cards in flight booking tasks. We share all agent logs, comprising 2.5B tokens of language model calls, to incentivize further research into agent behavior. By standardizing how the field evaluates agents and addressing common pitfalls in agent evaluation, we hope to shift the focus from agents that ace benchmarks to agents that work reliably in the real world. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_11977 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluation Kapoor, Sayash Stroebl, Benedikt Kirgis, Peter Nadgir, Nitya Siegel, Zachary S Wei, Boyi Xue, Tianci Chen, Ziru Chen, Felix Utpala, Saiteja Ndzomga, Franck Oruganty, Dheeraj Luskin, Sophie Liu, Kangheng Yu, Botao Arora, Amit Hahm, Dongyoon Trivedi, Harsh Sun, Huan Lee, Juyong Jin, Tengjun Mai, Yifan Zhou, Yifei Zhu, Yuxuan Bommasani, Rishi Kang, Daniel Song, Dawn Henderson, Peter Su, Yu Liang, Percy Narayanan, Arvind Artificial Intelligence Computation and Language AI agents have been developed for complex real-world tasks from coding to customer service. But AI agent evaluations suffer from many challenges that undermine our understanding of how well agents really work. We introduce the Holistic Agent Leaderboard (HAL) to address these challenges. We make three main contributions. First, we provide a standardized evaluation harness that orchestrates parallel evaluations across hundreds of VMs, reducing evaluation time from weeks to hours while eliminating common implementation bugs. Second, we conduct three-dimensional analysis spanning models, scaffolds, and benchmarks. We validate the harness by conducting 21,730 agent rollouts across 9 models and 9 benchmarks in coding, web navigation, science, and customer service with a total cost of about $40,000. Our analysis reveals surprising insights, such as higher reasoning effort reducing accuracy in the majority of runs. Third, we use LLM-aided log inspection to uncover previously unreported behaviors, such as searching for the benchmark on HuggingFace instead of solving a task, or misusing credit cards in flight booking tasks. We share all agent logs, comprising 2.5B tokens of language model calls, to incentivize further research into agent behavior. By standardizing how the field evaluates agents and addressing common pitfalls in agent evaluation, we hope to shift the focus from agents that ace benchmarks to agents that work reliably in the real world. |
| title | Holistic Agent Leaderboard: The Missing Infrastructure for AI Agent Evaluation |
| topic | Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2510.11977 |