Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2509.20998 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916970010509312 |
|---|---|
| author | Michelakis, Panagiotis Hadjiyiannis, Yiannis Stamoulis, Dimitrios |
| author_facet | Michelakis, Panagiotis Hadjiyiannis, Yiannis Stamoulis, Dimitrios |
| contents | Evaluating AI agents that solve real-world tasks through function-call sequences remains an open challenge. Existing agentic benchmarks often reduce evaluation to a binary judgment of the final state, overlooking critical aspects such as safety, efficiency, and intermediate correctness. We propose a framework based on deterministic finite automata (DFAs) that encodes tasks as sets of valid tool-use paths, enabling principled assessment of agent behavior in diverse world models. Building on this foundation, we introduce CORE, a suite of five metrics, namely Path Correctness, Path Correctness - Kendall's tau Composite, Prefix Criticality, Harmful-Call Rate, and Efficiency, that quantify alignment with expected execution patterns. Across diverse worlds, our method reveals important performance differences between agents that would otherwise appear equivalent under traditional final-state evaluation schemes. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_20998 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | CORE: Full-Path Evaluation of LLM Agents Beyond Final State Michelakis, Panagiotis Hadjiyiannis, Yiannis Stamoulis, Dimitrios Artificial Intelligence Evaluating AI agents that solve real-world tasks through function-call sequences remains an open challenge. Existing agentic benchmarks often reduce evaluation to a binary judgment of the final state, overlooking critical aspects such as safety, efficiency, and intermediate correctness. We propose a framework based on deterministic finite automata (DFAs) that encodes tasks as sets of valid tool-use paths, enabling principled assessment of agent behavior in diverse world models. Building on this foundation, we introduce CORE, a suite of five metrics, namely Path Correctness, Path Correctness - Kendall's tau Composite, Prefix Criticality, Harmful-Call Rate, and Efficiency, that quantify alignment with expected execution patterns. Across diverse worlds, our method reveals important performance differences between agents that would otherwise appear equivalent under traditional final-state evaluation schemes. |
| title | CORE: Full-Path Evaluation of LLM Agents Beyond Final State |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2509.20998 |