Holistic Evaluation and Failure Diagnosis of AI Agents
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2026
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866917495815798784 |
|---|---|
| author | Madvil, Netta Dym, Gilad Mecilati, Alon Dekel, Edo Liberman, Jonatan Brazilay, Rotem Schliesser, Liron Svidlo, Max Nir, Shai Shalom, Orel Friedman, Yaron Connack, David Rimon, Amos Tannor, Philip Chorev, Shir |
| author_facet | Madvil, Netta Dym, Gilad Mecilati, Alon Dekel, Edo Liberman, Jonatan Brazilay, Rotem Schliesser, Liron Svidlo, Max Nir, Shai Shalom, Orel Friedman, Yaron Connack, David Rimon, Amos Tannor, Philip Chorev, Shir |
| contents | AI agents execute complex multi-step processes, but current evaluation falls short: outcome metrics report success or failure without explaining why, and process-level approaches struggle to connect failure types to their precise locations within long, structured traces. We present a holistic agent evaluation framework that pairs top-down agent-level diagnosis with bottom-up span-level evaluation, decomposing analysis into independent per-span assessments. This decomposition scales to traces of arbitrary length and produces span-level rationales for each verdict. On the TRAIL benchmark, our framework achieves state-of-the-art results across all metrics on both GAIA and SWE-Bench, with relative gains over the strongest prior baselines of up to 38% on category F1, up to 3.5x on localization accuracy, and up to 12.5x on joint localization-categorization accuracy. Per-category analysis shows our framework leading in more error categories than any other evaluator. Notably, the same frontier model achieves several times higher localization accuracy when used inside our framework than as a monolithic judge over the full trace, showing that evaluation methodology, not model capability, is the bottleneck. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_14865 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Holistic Evaluation and Failure Diagnosis of AI Agents Madvil, Netta Dym, Gilad Mecilati, Alon Dekel, Edo Liberman, Jonatan Brazilay, Rotem Schliesser, Liron Svidlo, Max Nir, Shai Shalom, Orel Friedman, Yaron Connack, David Rimon, Amos Tannor, Philip Chorev, Shir Artificial Intelligence Computation and Language AI agents execute complex multi-step processes, but current evaluation falls short: outcome metrics report success or failure without explaining why, and process-level approaches struggle to connect failure types to their precise locations within long, structured traces. We present a holistic agent evaluation framework that pairs top-down agent-level diagnosis with bottom-up span-level evaluation, decomposing analysis into independent per-span assessments. This decomposition scales to traces of arbitrary length and produces span-level rationales for each verdict. On the TRAIL benchmark, our framework achieves state-of-the-art results across all metrics on both GAIA and SWE-Bench, with relative gains over the strongest prior baselines of up to 38% on category F1, up to 3.5x on localization accuracy, and up to 12.5x on joint localization-categorization accuracy. Per-category analysis shows our framework leading in more error categories than any other evaluator. Notably, the same frontier model achieves several times higher localization accuracy when used inside our framework than as a monolithic judge over the full trace, showing that evaluation methodology, not model capability, is the bottleneck. |
| title | Holistic Evaluation and Failure Diagnosis of AI Agents |
| topic | Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2605.14865 |