AgentArcEval: An Architecture Evaluation Method for Foundation Model based Agents
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909867125506048 |
|---|---|
| author | Lu, Qinghua Zhao, Dehai Liu, Yue Zhang, Hao Zhu, Liming Xu, Xiwei Shi, Angela Tan, Tristan Kazman, Rick |
| author_facet | Lu, Qinghua Zhao, Dehai Liu, Yue Zhang, Hao Zhu, Liming Xu, Xiwei Shi, Angela Tan, Tristan Kazman, Rick |
| contents | The emergence of foundation models (FMs) has enabled the development of highly capable and autonomous agents, unlocking new application opportunities across a wide range of domains. Evaluating the architecture of agents is particularly important as the architectural decisions significantly impact the quality attributes of agents given their unique characteristics, including compound architecture, autonomous and non-deterministic behaviour, and continuous evolution. However, these traditional methods fall short in addressing the evaluation needs of agent architecture due to the unique characteristics of these agents. Therefore, in this paper, we present AgentArcEval, a novel agent architecture evaluation method designed specially to address the complexities of FM-based agent architecture and its evaluation. Moreover, we present a catalogue of agent-specific general scenarios, which serves as a guide for generating concrete scenarios to design and evaluate the agent architecture. We demonstrate the usefulness of AgentArcEval and the catalogue through a case study on the architecture evaluation of a real-world tax copilot, named Luna. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_21031 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | AgentArcEval: An Architecture Evaluation Method for Foundation Model based Agents Lu, Qinghua Zhao, Dehai Liu, Yue Zhang, Hao Zhu, Liming Xu, Xiwei Shi, Angela Tan, Tristan Kazman, Rick Software Engineering Artificial Intelligence The emergence of foundation models (FMs) has enabled the development of highly capable and autonomous agents, unlocking new application opportunities across a wide range of domains. Evaluating the architecture of agents is particularly important as the architectural decisions significantly impact the quality attributes of agents given their unique characteristics, including compound architecture, autonomous and non-deterministic behaviour, and continuous evolution. However, these traditional methods fall short in addressing the evaluation needs of agent architecture due to the unique characteristics of these agents. Therefore, in this paper, we present AgentArcEval, a novel agent architecture evaluation method designed specially to address the complexities of FM-based agent architecture and its evaluation. Moreover, we present a catalogue of agent-specific general scenarios, which serves as a guide for generating concrete scenarios to design and evaluate the agent architecture. We demonstrate the usefulness of AgentArcEval and the catalogue through a case study on the architecture evaluation of a real-world tax copilot, named Luna. |
| title | AgentArcEval: An Architecture Evaluation Method for Foundation Model based Agents |
| topic | Software Engineering Artificial Intelligence |
| url | https://arxiv.org/abs/2510.21031 |