UFO2: The Desktop AgentOS
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917998490550272 |
|---|---|
| author | Zhang, Chaoyun Huang, He Ni, Chiming Mu, Jian Qin, Si He, Shilin Wang, Lu Yang, Fangkai Zhao, Pu Du, Chao Li, Liqun Kang, Yu Jiang, Zhao Zheng, Suzhen Wang, Rujia Qian, Jiaxu Ma, Minghua Lou, Jian-Guang Lin, Qingwei Rajmohan, Saravan Zhang, Dongmei |
| author_facet | Zhang, Chaoyun Huang, He Ni, Chiming Mu, Jian Qin, Si He, Shilin Wang, Lu Yang, Fangkai Zhao, Pu Du, Chao Li, Liqun Kang, Yu Jiang, Zhao Zheng, Suzhen Wang, Rujia Qian, Jiaxu Ma, Minghua Lou, Jian-Guang Lin, Qingwei Rajmohan, Saravan Zhang, Dongmei |
| contents | Recent Computer-Using Agents (CUAs), powered by multimodal large language models (LLMs), offer a promising direction for automating complex desktop workflows through natural language. However, most existing CUAs remain conceptual prototypes, hindered by shallow OS integration, fragile screenshot-based interaction, and disruptive execution.
We present UFO2, a multiagent AgentOS for Windows desktops that elevates CUAs into practical, system-level automation. UFO2 features a centralized HostAgent for task decomposition and coordination, alongside a collection of application-specialized AppAgent equipped with native APIs, domain-specific knowledge, and a unified GUI--API action layer. This architecture enables robust task execution while preserving modularity and extensibility. A hybrid control detection pipeline fuses Windows UI Automation (UIA) with vision-based parsing to support diverse interface styles. Runtime efficiency is further enhanced through speculative multi-action planning, reducing per-step LLM overhead. Finally, a Picture-in-Picture (PiP) interface enables automation within an isolated virtual desktop, allowing agents and users to operate concurrently without interference.
We evaluate UFO2 across over 20 real-world Windows applications, demonstrating substantial improvements in robustness and execution accuracy over prior CUAs. Our results show that deep OS integration unlocks a scalable path toward reliable, user-aligned desktop automation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_14603 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | UFO2: The Desktop AgentOS Zhang, Chaoyun Huang, He Ni, Chiming Mu, Jian Qin, Si He, Shilin Wang, Lu Yang, Fangkai Zhao, Pu Du, Chao Li, Liqun Kang, Yu Jiang, Zhao Zheng, Suzhen Wang, Rujia Qian, Jiaxu Ma, Minghua Lou, Jian-Guang Lin, Qingwei Rajmohan, Saravan Zhang, Dongmei Artificial Intelligence Human-Computer Interaction Operating Systems Recent Computer-Using Agents (CUAs), powered by multimodal large language models (LLMs), offer a promising direction for automating complex desktop workflows through natural language. However, most existing CUAs remain conceptual prototypes, hindered by shallow OS integration, fragile screenshot-based interaction, and disruptive execution. We present UFO2, a multiagent AgentOS for Windows desktops that elevates CUAs into practical, system-level automation. UFO2 features a centralized HostAgent for task decomposition and coordination, alongside a collection of application-specialized AppAgent equipped with native APIs, domain-specific knowledge, and a unified GUI--API action layer. This architecture enables robust task execution while preserving modularity and extensibility. A hybrid control detection pipeline fuses Windows UI Automation (UIA) with vision-based parsing to support diverse interface styles. Runtime efficiency is further enhanced through speculative multi-action planning, reducing per-step LLM overhead. Finally, a Picture-in-Picture (PiP) interface enables automation within an isolated virtual desktop, allowing agents and users to operate concurrently without interference. We evaluate UFO2 across over 20 real-world Windows applications, demonstrating substantial improvements in robustness and execution accuracy over prior CUAs. Our results show that deep OS integration unlocks a scalable path toward reliable, user-aligned desktop automation. |
| title | UFO2: The Desktop AgentOS |
| topic | Artificial Intelligence Human-Computer Interaction Operating Systems |
| url | https://arxiv.org/abs/2504.14603 |