Salvato in:
| Autori principali: | , , , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | https://arxiv.org/abs/2509.24002 |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866916974948253696 |
|---|---|
| author | Wu, Zijian Liu, Xiangyan Zhang, Xinyuan Chen, Lingjun Meng, Fanqing Du, Lingxiao Zhao, Yiran Zhang, Fanshi Ye, Yaoqi Wang, Jiawei Wang, Zirui Ni, Jinjie Yang, Yufan Xu, Arvin Shieh, Michael Qizhe |
| author_facet | Wu, Zijian Liu, Xiangyan Zhang, Xinyuan Chen, Lingjun Meng, Fanqing Du, Lingxiao Zhao, Yiran Zhang, Fanshi Ye, Yaoqi Wang, Jiawei Wang, Zirui Ni, Jinjie Yang, Yufan Xu, Arvin Shieh, Michael Qizhe |
| contents | MCP standardizes how LLMs interact with external systems, forming the foundation for general agents. However, existing MCP benchmarks remain narrow in scope: they focus on read-heavy tasks or tasks with limited interaction depth, and fail to capture the complexity and realism of real-world workflows. To address this gap, we propose MCPMark, a benchmark designed to evaluate MCP use in a more realistic and comprehensive manner. It consists of $127$ high-quality tasks collaboratively created by domain experts and AI agents. Each task begins with a curated initial state and includes a programmatic script for automatic verification. These tasks demand richer and more diverse interactions with the environment, involving a broad range of create, read, update, and delete (CRUD) operations. We conduct a comprehensive evaluation of cutting-edge LLMs using a minimal agent framework that operates in a tool-calling loop. Empirical results show that the best-performing model, gpt-5-medium, reaches only $52.56$\% pass@1 and $33.86$\% pass^4, while other widely regarded strong models, including claude-sonnet-4 and o3, fall below $30$\% pass@1 and $15$\% pass^4. On average, LLMs require $16.2$ execution turns and $17.4$ tool calls per task, significantly surpassing those in previous MCP benchmarks and highlighting the stress-testing nature of MCPMark. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_24002 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | MCPMark: A Benchmark for Stress-Testing Realistic and Comprehensive MCP Use Wu, Zijian Liu, Xiangyan Zhang, Xinyuan Chen, Lingjun Meng, Fanqing Du, Lingxiao Zhao, Yiran Zhang, Fanshi Ye, Yaoqi Wang, Jiawei Wang, Zirui Ni, Jinjie Yang, Yufan Xu, Arvin Shieh, Michael Qizhe Computation and Language Artificial Intelligence MCP standardizes how LLMs interact with external systems, forming the foundation for general agents. However, existing MCP benchmarks remain narrow in scope: they focus on read-heavy tasks or tasks with limited interaction depth, and fail to capture the complexity and realism of real-world workflows. To address this gap, we propose MCPMark, a benchmark designed to evaluate MCP use in a more realistic and comprehensive manner. It consists of $127$ high-quality tasks collaboratively created by domain experts and AI agents. Each task begins with a curated initial state and includes a programmatic script for automatic verification. These tasks demand richer and more diverse interactions with the environment, involving a broad range of create, read, update, and delete (CRUD) operations. We conduct a comprehensive evaluation of cutting-edge LLMs using a minimal agent framework that operates in a tool-calling loop. Empirical results show that the best-performing model, gpt-5-medium, reaches only $52.56$\% pass@1 and $33.86$\% pass^4, while other widely regarded strong models, including claude-sonnet-4 and o3, fall below $30$\% pass@1 and $15$\% pass^4. On average, LLMs require $16.2$ execution turns and $17.4$ tool calls per task, significantly surpassing those in previous MCP benchmarks and highlighting the stress-testing nature of MCPMark. |
| title | MCPMark: A Benchmark for Stress-Testing Realistic and Comprehensive MCP Use |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2509.24002 |