MAESTRO: Multi-Agent Evaluation Suite for Testing, Reliability, and Observability
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2026
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866911350377152512 |
|---|---|
| author | Ma, Tie Chen, Yixi Anand, Vaastav Cornacchia, Alessandro Faustino, Amândio R. Liu, Guanheng Zhang, Shan Luo, Hongbin Fahmy, Suhaib A. Qazi, Zafar A. Canini, Marco |
| author_facet | Ma, Tie Chen, Yixi Anand, Vaastav Cornacchia, Alessandro Faustino, Amândio R. Liu, Guanheng Zhang, Shan Luo, Hongbin Fahmy, Suhaib A. Qazi, Zafar A. Canini, Marco |
| contents | We present MAESTRO, an evaluation suite for the testing, reliability, and observability of LLM-based MAS. MAESTRO standardizes MAS configuration and execution through a unified interface, supports integrating both native and third-party MAS via a repository of examples and lightweight adapters, and exports framework-agnostic execution traces together with system-level signals (e.g., latency, cost, and failures). We instantiate MAESTRO with 12 representative MAS spanning popular agentic frameworks and interaction patterns, and conduct controlled experiments across repeated runs, backend models, and tool configurations. Our case studies show that MAS executions can be structurally stable yet temporally variable, leading to substantial run-to-run variance in performance and reliability. We further find that MAS architecture is the dominant driver of resource profiles, reproducibility, and cost-latency-accuracy trade-off, often outweighing changes in backend models or tool settings. Overall, MAESTRO enables systematic evaluation and provides empirical guidance for designing and optimizing agentic systems. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_00481 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | MAESTRO: Multi-Agent Evaluation Suite for Testing, Reliability, and Observability Ma, Tie Chen, Yixi Anand, Vaastav Cornacchia, Alessandro Faustino, Amândio R. Liu, Guanheng Zhang, Shan Luo, Hongbin Fahmy, Suhaib A. Qazi, Zafar A. Canini, Marco Networking and Internet Architecture Artificial Intelligence We present MAESTRO, an evaluation suite for the testing, reliability, and observability of LLM-based MAS. MAESTRO standardizes MAS configuration and execution through a unified interface, supports integrating both native and third-party MAS via a repository of examples and lightweight adapters, and exports framework-agnostic execution traces together with system-level signals (e.g., latency, cost, and failures). We instantiate MAESTRO with 12 representative MAS spanning popular agentic frameworks and interaction patterns, and conduct controlled experiments across repeated runs, backend models, and tool configurations. Our case studies show that MAS executions can be structurally stable yet temporally variable, leading to substantial run-to-run variance in performance and reliability. We further find that MAS architecture is the dominant driver of resource profiles, reproducibility, and cost-latency-accuracy trade-off, often outweighing changes in backend models or tool settings. Overall, MAESTRO enables systematic evaluation and provides empirical guidance for designing and optimizing agentic systems. |
| title | MAESTRO: Multi-Agent Evaluation Suite for Testing, Reliability, and Observability |
| topic | Networking and Internet Architecture Artificial Intelligence |
| url | https://arxiv.org/abs/2601.00481 |