Survey on Evaluation of LLM-based Agents
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866911617414856704 |
|---|---|
| author | Yehudai, Asaf Eden, Lilach Li, Alan Uziel, Guy Zhao, Yilun Bar-Haim, Roy Cohan, Arman Shmueli-Scheuer, Michal |
| author_facet | Yehudai, Asaf Eden, Lilach Li, Alan Uziel, Guy Zhao, Yilun Bar-Haim, Roy Cohan, Arman Shmueli-Scheuer, Michal |
| contents | LLM-based agents represent a paradigm shift in AI, enabling autonomous systems to plan, reason, and use tools while interacting with dynamic environments. This paper provides the first comprehensive survey of evaluation methods for these increasingly capable agents. We analyze the field of agent evaluation across five perspectives: (1) Core LLM capabilities needed for agentic workflows, like planning, and tool use; (2) Application-specific benchmarks such as web and SWE agents; (3) Evaluation of generalist agents; (4) Analysis of agent benchmarks' core dimensions; and (5) Evaluation frameworks and tools for agent developers. Our analysis reveals current trends, including a shift toward more realistic, challenging evaluations with continuously updated benchmarks. We also identify critical gaps that future research must address, particularly in assessing cost-efficiency, safety, and robustness, and in developing fine-grained, scalable evaluation methods. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_16416 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Survey on Evaluation of LLM-based Agents Yehudai, Asaf Eden, Lilach Li, Alan Uziel, Guy Zhao, Yilun Bar-Haim, Roy Cohan, Arman Shmueli-Scheuer, Michal Artificial Intelligence Computation and Language Machine Learning LLM-based agents represent a paradigm shift in AI, enabling autonomous systems to plan, reason, and use tools while interacting with dynamic environments. This paper provides the first comprehensive survey of evaluation methods for these increasingly capable agents. We analyze the field of agent evaluation across five perspectives: (1) Core LLM capabilities needed for agentic workflows, like planning, and tool use; (2) Application-specific benchmarks such as web and SWE agents; (3) Evaluation of generalist agents; (4) Analysis of agent benchmarks' core dimensions; and (5) Evaluation frameworks and tools for agent developers. Our analysis reveals current trends, including a shift toward more realistic, challenging evaluations with continuously updated benchmarks. We also identify critical gaps that future research must address, particularly in assessing cost-efficiency, safety, and robustness, and in developing fine-grained, scalable evaluation methods. |
| title | Survey on Evaluation of LLM-based Agents |
| topic | Artificial Intelligence Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2503.16416 |