Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2507.02825 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915433050800128 |
|---|---|
| author | Zhu, Yuxuan Jin, Tengjun Pruksachatkun, Yada Zhang, Andy Liu, Shu Cui, Sasha Kapoor, Sayash Longpre, Shayne Meng, Kevin Weiss, Rebecca Barez, Fazl Gupta, Rahul Dhamala, Jwala Merizian, Jacob Giulianelli, Mario Coppock, Harry Ududec, Cozmin Sekhon, Jasjeet Steinhardt, Jacob Kellermann, Antony Schwettmann, Sarah Zaharia, Matei Stoica, Ion Liang, Percy Kang, Daniel |
| author_facet | Zhu, Yuxuan Jin, Tengjun Pruksachatkun, Yada Zhang, Andy Liu, Shu Cui, Sasha Kapoor, Sayash Longpre, Shayne Meng, Kevin Weiss, Rebecca Barez, Fazl Gupta, Rahul Dhamala, Jwala Merizian, Jacob Giulianelli, Mario Coppock, Harry Ududec, Cozmin Sekhon, Jasjeet Steinhardt, Jacob Kellermann, Antony Schwettmann, Sarah Zaharia, Matei Stoica, Ion Liang, Percy Kang, Daniel |
| contents | Benchmarks are essential for quantitatively tracking progress in AI. As AI agents become increasingly capable, researchers and practitioners have introduced agentic benchmarks to evaluate agents on complex, real-world tasks. These benchmarks typically measure agent capabilities by evaluating task outcomes via specific reward designs. However, we show that many agentic benchmarks have issues in task setup or reward design. For example, SWE-bench Verified uses insufficient test cases, while TAU-bench counts empty responses as successful. Such issues can lead to under- or overestimation of agents' performance by up to 100% in relative terms. To make agentic evaluation rigorous, we introduce the Agentic Benchmark Checklist (ABC), a set of guidelines that we synthesized from our benchmark-building experience, a survey of best practices, and previously reported issues. When applied to CVE-Bench, a benchmark with a particularly complex evaluation design, ABC reduces the performance overestimation by 33%. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_02825 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Establishing Best Practices for Building Rigorous Agentic Benchmarks Zhu, Yuxuan Jin, Tengjun Pruksachatkun, Yada Zhang, Andy Liu, Shu Cui, Sasha Kapoor, Sayash Longpre, Shayne Meng, Kevin Weiss, Rebecca Barez, Fazl Gupta, Rahul Dhamala, Jwala Merizian, Jacob Giulianelli, Mario Coppock, Harry Ududec, Cozmin Sekhon, Jasjeet Steinhardt, Jacob Kellermann, Antony Schwettmann, Sarah Zaharia, Matei Stoica, Ion Liang, Percy Kang, Daniel Artificial Intelligence A.1; I.2.m Benchmarks are essential for quantitatively tracking progress in AI. As AI agents become increasingly capable, researchers and practitioners have introduced agentic benchmarks to evaluate agents on complex, real-world tasks. These benchmarks typically measure agent capabilities by evaluating task outcomes via specific reward designs. However, we show that many agentic benchmarks have issues in task setup or reward design. For example, SWE-bench Verified uses insufficient test cases, while TAU-bench counts empty responses as successful. Such issues can lead to under- or overestimation of agents' performance by up to 100% in relative terms. To make agentic evaluation rigorous, we introduce the Agentic Benchmark Checklist (ABC), a set of guidelines that we synthesized from our benchmark-building experience, a survey of best practices, and previously reported issues. When applied to CVE-Bench, a benchmark with a particularly complex evaluation design, ABC reduces the performance overestimation by 33%. |
| title | Establishing Best Practices for Building Rigorous Agentic Benchmarks |
| topic | Artificial Intelligence A.1; I.2.m |
| url | https://arxiv.org/abs/2507.02825 |