HumanMCP: A Human-Like Query Dataset for Evaluating MCP Tool Retrieval Performance
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866911472807837696 |
|---|---|
| author | Laddha, Shubh Changbencharoen, Lucas Kuptivej, Win Shringla, Surya Vaidheeswaran, Archana Bhaskar, Yash |
| author_facet | Laddha, Shubh Changbencharoen, Lucas Kuptivej, Win Shringla, Surya Vaidheeswaran, Archana Bhaskar, Yash |
| contents | Model Context Protocol (MCP) servers contain a collection of thousands of open-source standardized tools, linking LLMs to external systems; however, existing datasets and benchmarks lack realistic, human-like user queries, remaining a critical gap in evaluating the tool usage and ecosystems of MCP servers. Existing datasets often do contain tool descriptions but fail to represent how different users portray their requests, leading to poor generalization and inflated reliability of certain benchmarks. This paper introduces the first large-scale MCP dataset featuring diverse, high-quality diverse user queries generated specifically to match 2800 tools across 308 MCP servers, developing on the MCP Zero dataset. Each tool is paired with multiple unique user personas that we have generated, to capture varying levels of user intent ranging from precise task requests, and ambiguous, exploratory commands, reflecting the complexity of real-world interaction patterns. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_23367 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | HumanMCP: A Human-Like Query Dataset for Evaluating MCP Tool Retrieval Performance Laddha, Shubh Changbencharoen, Lucas Kuptivej, Win Shringla, Surya Vaidheeswaran, Archana Bhaskar, Yash Artificial Intelligence Information Retrieval 68T01, 68T50 I.2.11; H.3.3; I.2.7 Model Context Protocol (MCP) servers contain a collection of thousands of open-source standardized tools, linking LLMs to external systems; however, existing datasets and benchmarks lack realistic, human-like user queries, remaining a critical gap in evaluating the tool usage and ecosystems of MCP servers. Existing datasets often do contain tool descriptions but fail to represent how different users portray their requests, leading to poor generalization and inflated reliability of certain benchmarks. This paper introduces the first large-scale MCP dataset featuring diverse, high-quality diverse user queries generated specifically to match 2800 tools across 308 MCP servers, developing on the MCP Zero dataset. Each tool is paired with multiple unique user personas that we have generated, to capture varying levels of user intent ranging from precise task requests, and ambiguous, exploratory commands, reflecting the complexity of real-world interaction patterns. |
| title | HumanMCP: A Human-Like Query Dataset for Evaluating MCP Tool Retrieval Performance |
| topic | Artificial Intelligence Information Retrieval 68T01, 68T50 I.2.11; H.3.3; I.2.7 |
| url | https://arxiv.org/abs/2602.23367 |