MASEval: Extending Multi-Agent Evaluation from Models to Systems
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915848778678272 |
|---|---|
| author | Emde, Cornelius Rubinstein, Alexander Goel, Anmol Heakl, Ahmed Yun, Sangdoo Oh, Seong Joon Gubri, Martin |
| author_facet | Emde, Cornelius Rubinstein, Alexander Goel, Anmol Heakl, Ahmed Yun, Sangdoo Oh, Seong Joon Gubri, Martin |
| contents | The rapid adoption of LLM-based agentic systems has produced a rich ecosystem of frameworks (smolagents, LangGraph, AutoGen, CAMEL, LlamaIndex, i.a.). Yet existing benchmarks are model-centric: they fix the agentic setup and do not compare other system components. We argue that implementation decisions substantially impact performance, including choices such as topology, orchestration logic, and error handling. MASEval addresses this evaluation gap with a framework-agnostic library that treats the entire system as the unit of analysis. Through a systematic system-level comparison across 3 benchmarks, 3 models, and 3 frameworks, we find that framework choice matters as much as model choice. MASEval allows researchers to explore all components of agentic systems, opening new avenues for principled system design, and practitioners to identify the best implementation for their use case.
MASEval is available under the MIT licence https://github.com/parameterlab/MASEval. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_08835 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | MASEval: Extending Multi-Agent Evaluation from Models to Systems Emde, Cornelius Rubinstein, Alexander Goel, Anmol Heakl, Ahmed Yun, Sangdoo Oh, Seong Joon Gubri, Martin Artificial Intelligence Computation and Language Machine Learning The rapid adoption of LLM-based agentic systems has produced a rich ecosystem of frameworks (smolagents, LangGraph, AutoGen, CAMEL, LlamaIndex, i.a.). Yet existing benchmarks are model-centric: they fix the agentic setup and do not compare other system components. We argue that implementation decisions substantially impact performance, including choices such as topology, orchestration logic, and error handling. MASEval addresses this evaluation gap with a framework-agnostic library that treats the entire system as the unit of analysis. Through a systematic system-level comparison across 3 benchmarks, 3 models, and 3 frameworks, we find that framework choice matters as much as model choice. MASEval allows researchers to explore all components of agentic systems, opening new avenues for principled system design, and practitioners to identify the best implementation for their use case. MASEval is available under the MIT licence https://github.com/parameterlab/MASEval. |
| title | MASEval: Extending Multi-Agent Evaluation from Models to Systems |
| topic | Artificial Intelligence Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2603.08835 |