Saved in:
| Main Authors: | , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2506.07672 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912419645751296 |
|---|---|
| author | Yan, Yunhe Wang, Shihe Du, Jiajun Yang, Yexuan Shan, Yuxuan Qiu, Qichen Jia, Xianqing Wang, Xinge Yuan, Xin Han, Xu Qin, Mao Chen, Yinxiao Peng, Chen Wang, Shangguang Xu, Mengwei |
| author_facet | Yan, Yunhe Wang, Shihe Du, Jiajun Yang, Yexuan Shan, Yuxuan Qiu, Qichen Jia, Xianqing Wang, Xinge Yuan, Xin Han, Xu Qin, Mao Chen, Yinxiao Peng, Chen Wang, Shangguang Xu, Mengwei |
| contents | (M)LLM-powered computer use agents (CUA) are emerging as a transformative technique to automate human-computer interaction. However, existing CUA benchmarks predominantly target GUI agents, whose evaluation methods are susceptible to UI changes and ignore function interactions exposed by application APIs, e.g., Model Context Protocol (MCP). To this end, we propose MCPWorld, the first automatic CUA testbed for API, GUI, and API-GUI hybrid agents. A key principle of MCPWorld is the use of "white-box apps", i.e., those with source code availability and can be revised/re-compiled as needed (e.g., adding MCP support), with two notable advantages:
(1) It greatly broadens the design space of CUA, such as what and how the app features to be exposed/extracted as CUA-callable APIs.
(2) It allows MCPWorld to programmatically verify task completion by directly monitoring application behavior through techniques like dynamic code instrumentation, offering robust, accurate CUA evaluation decoupled from specific agent implementations or UI states.
Currently, MCPWorld includes 201 well curated and annotated user tasks, covering diversified use cases and difficulty levels. MCPWorld is also fully containerized with GPU acceleration support for flexible adoption on different OS/hardware environments. Our preliminary experiments, using a representative LLM-powered CUA framework, achieve 75.12% task completion accuracy, simultaneously providing initial evidence on the practical effectiveness of agent automation leveraging MCP. Overall, we anticipate MCPWorld to facilitate and standardize the benchmarking of next-generation computer use agents that can leverage rich external tools. Our code and dataset are publicly available at https://github.com/SAAgent/MCPWorld. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_07672 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | MCPWorld: A Unified Benchmarking Testbed for API, GUI, and Hybrid Computer Use Agents Yan, Yunhe Wang, Shihe Du, Jiajun Yang, Yexuan Shan, Yuxuan Qiu, Qichen Jia, Xianqing Wang, Xinge Yuan, Xin Han, Xu Qin, Mao Chen, Yinxiao Peng, Chen Wang, Shangguang Xu, Mengwei Artificial Intelligence (M)LLM-powered computer use agents (CUA) are emerging as a transformative technique to automate human-computer interaction. However, existing CUA benchmarks predominantly target GUI agents, whose evaluation methods are susceptible to UI changes and ignore function interactions exposed by application APIs, e.g., Model Context Protocol (MCP). To this end, we propose MCPWorld, the first automatic CUA testbed for API, GUI, and API-GUI hybrid agents. A key principle of MCPWorld is the use of "white-box apps", i.e., those with source code availability and can be revised/re-compiled as needed (e.g., adding MCP support), with two notable advantages: (1) It greatly broadens the design space of CUA, such as what and how the app features to be exposed/extracted as CUA-callable APIs. (2) It allows MCPWorld to programmatically verify task completion by directly monitoring application behavior through techniques like dynamic code instrumentation, offering robust, accurate CUA evaluation decoupled from specific agent implementations or UI states. Currently, MCPWorld includes 201 well curated and annotated user tasks, covering diversified use cases and difficulty levels. MCPWorld is also fully containerized with GPU acceleration support for flexible adoption on different OS/hardware environments. Our preliminary experiments, using a representative LLM-powered CUA framework, achieve 75.12% task completion accuracy, simultaneously providing initial evidence on the practical effectiveness of agent automation leveraging MCP. Overall, we anticipate MCPWorld to facilitate and standardize the benchmarking of next-generation computer use agents that can leverage rich external tools. Our code and dataset are publicly available at https://github.com/SAAgent/MCPWorld. |
| title | MCPWorld: A Unified Benchmarking Testbed for API, GUI, and Hybrid Computer Use Agents |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2506.07672 |