ETOM: A Five-Level Benchmark for Evaluating Tool Orchestration within the MCP Ecosystem
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866912831141576704 |
|---|---|
| author | Dong, Jia-Kai Huang, I-Wei Wu, Chun-Tin Tsai, Yi-Tien |
| author_facet | Dong, Jia-Kai Huang, I-Wei Wu, Chun-Tin Tsai, Yi-Tien |
| contents | We introduce ETOM, a five-level benchmark for evaluating multi-hop, end-to-end tool orchestration by LLM agents within a hierarchical Model-Context Protocol (MCP) ecosystem. Existing benchmarks often assess tools in isolation, overlooking challenges such as functional overlap and cross-server orchestration, which can lead to overly optimistic evaluations. ETOM addresses these gaps by constructing ground truth through "equal function sets", enabling objective metrics such as F1 score and reducing reliance on LLM-as-a-judge evaluation. Its five-level curriculum systematically tests agent capabilities, from single-tool orchestration to complex cross-server planning, as well as robustness to out-of-scope requests. Experiments reveal that rigid hierarchies can hinder performance without co-designed strategies, and even state-of-the-art agents exhibit systemic weaknesses in robustness. ETOM provides a diagnostic framework to expose these limitations and guide the development of more capable and efficient tool-using agents. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_19423 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | ETOM: A Five-Level Benchmark for Evaluating Tool Orchestration within the MCP Ecosystem Dong, Jia-Kai Huang, I-Wei Wu, Chun-Tin Tsai, Yi-Tien Artificial Intelligence I.2.0; I.2.1; I.2.4 We introduce ETOM, a five-level benchmark for evaluating multi-hop, end-to-end tool orchestration by LLM agents within a hierarchical Model-Context Protocol (MCP) ecosystem. Existing benchmarks often assess tools in isolation, overlooking challenges such as functional overlap and cross-server orchestration, which can lead to overly optimistic evaluations. ETOM addresses these gaps by constructing ground truth through "equal function sets", enabling objective metrics such as F1 score and reducing reliance on LLM-as-a-judge evaluation. Its five-level curriculum systematically tests agent capabilities, from single-tool orchestration to complex cross-server planning, as well as robustness to out-of-scope requests. Experiments reveal that rigid hierarchies can hinder performance without co-designed strategies, and even state-of-the-art agents exhibit systemic weaknesses in robustness. ETOM provides a diagnostic framework to expose these limitations and guide the development of more capable and efficient tool-using agents. |
| title | ETOM: A Five-Level Benchmark for Evaluating Tool Orchestration within the MCP Ecosystem |
| topic | Artificial Intelligence I.2.0; I.2.1; I.2.4 |
| url | https://arxiv.org/abs/2510.19423 |