Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2403.03186 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909237615001600 |
|---|---|
| author | Tan, Weihao Zhang, Wentao Xu, Xinrun Xia, Haochong Ding, Ziluo Li, Boyu Zhou, Bohan Yue, Junpeng Jiang, Jiechuan Li, Yewen An, Ruyi Qin, Molei Zong, Chuqiao Zheng, Longtao Wu, Yujie Chai, Xiaoqiang Bi, Yifei Xie, Tianbao Gu, Pengjie Li, Xiyun Zhang, Ceyao Tian, Long Wang, Chaojie Wang, Xinrun Karlsson, Börje F. An, Bo Yan, Shuicheng Lu, Zongqing |
| author_facet | Tan, Weihao Zhang, Wentao Xu, Xinrun Xia, Haochong Ding, Ziluo Li, Boyu Zhou, Bohan Yue, Junpeng Jiang, Jiechuan Li, Yewen An, Ruyi Qin, Molei Zong, Chuqiao Zheng, Longtao Wu, Yujie Chai, Xiaoqiang Bi, Yifei Xie, Tianbao Gu, Pengjie Li, Xiyun Zhang, Ceyao Tian, Long Wang, Chaojie Wang, Xinrun Karlsson, Börje F. An, Bo Yan, Shuicheng Lu, Zongqing |
| contents | Despite the success in specific scenarios, existing foundation agents still struggle to generalize across various virtual scenarios, mainly due to the dramatically different encapsulations of environments with manually designed observation and action spaces. To handle this issue, we propose the General Computer Control (GCC) setting to restrict foundation agents to interact with software through the most unified and standardized interface, i.e., using screenshots as input and keyboard and mouse actions as output. We introduce Cradle, a modular and flexible LMM-powered framework, as a preliminary attempt towards GCC. Enhanced by six key modules, Cradle can understand input screenshots and output executable code for low-level keyboard and mouse control after high-level planning, so that Cradle can interact with any software and complete long-horizon complex tasks without relying on any built-in APIs. Experimental results show that Cradle exhibits remarkable generalizability and impressive performance across four previously unexplored commercial video games, five software applications, and a comprehensive benchmark, OSWorld. Cradle is the first to enable foundation agents to follow the main storyline and complete 40-minute-long real missions in the complex AAA game Red Dead Redemption 2 (RDR2). Cradle can also create a city of a thousand people in Cities: Skylines, farm and harvest parsnips in Stardew Valley, and trade and bargain with a maximal weekly total profit of 87% in Dealer's Life 2. Cradle can not only operate daily software, like Chrome, Outlook, and Feishu, but also edit images and videos using Meitu and CapCut. Cradle greatly extends the reach of foundation agents by enabling the easy conversion of any software, especially complex games, into benchmarks to evaluate agents' various abilities and facilitate further data collection, thus paving the way for generalist agents. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_03186 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Cradle: Empowering Foundation Agents Towards General Computer Control Tan, Weihao Zhang, Wentao Xu, Xinrun Xia, Haochong Ding, Ziluo Li, Boyu Zhou, Bohan Yue, Junpeng Jiang, Jiechuan Li, Yewen An, Ruyi Qin, Molei Zong, Chuqiao Zheng, Longtao Wu, Yujie Chai, Xiaoqiang Bi, Yifei Xie, Tianbao Gu, Pengjie Li, Xiyun Zhang, Ceyao Tian, Long Wang, Chaojie Wang, Xinrun Karlsson, Börje F. An, Bo Yan, Shuicheng Lu, Zongqing Artificial Intelligence Despite the success in specific scenarios, existing foundation agents still struggle to generalize across various virtual scenarios, mainly due to the dramatically different encapsulations of environments with manually designed observation and action spaces. To handle this issue, we propose the General Computer Control (GCC) setting to restrict foundation agents to interact with software through the most unified and standardized interface, i.e., using screenshots as input and keyboard and mouse actions as output. We introduce Cradle, a modular and flexible LMM-powered framework, as a preliminary attempt towards GCC. Enhanced by six key modules, Cradle can understand input screenshots and output executable code for low-level keyboard and mouse control after high-level planning, so that Cradle can interact with any software and complete long-horizon complex tasks without relying on any built-in APIs. Experimental results show that Cradle exhibits remarkable generalizability and impressive performance across four previously unexplored commercial video games, five software applications, and a comprehensive benchmark, OSWorld. Cradle is the first to enable foundation agents to follow the main storyline and complete 40-minute-long real missions in the complex AAA game Red Dead Redemption 2 (RDR2). Cradle can also create a city of a thousand people in Cities: Skylines, farm and harvest parsnips in Stardew Valley, and trade and bargain with a maximal weekly total profit of 87% in Dealer's Life 2. Cradle can not only operate daily software, like Chrome, Outlook, and Feishu, but also edit images and videos using Meitu and CapCut. Cradle greatly extends the reach of foundation agents by enabling the easy conversion of any software, especially complex games, into benchmarks to evaluate agents' various abilities and facilitate further data collection, thus paving the way for generalist agents. |
| title | Cradle: Empowering Foundation Agents Towards General Computer Control |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2403.03186 |