OS-Copilot: Towards Generalist Computer Agents with Self-Improvement
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866911777389805568 |
|---|---|
| author | Wu, Zhiyong Han, Chengcheng Ding, Zichen Weng, Zhenmin Liu, Zhoumianze Yao, Shunyu Yu, Tao Kong, Lingpeng |
| author_facet | Wu, Zhiyong Han, Chengcheng Ding, Zichen Weng, Zhenmin Liu, Zhoumianze Yao, Shunyu Yu, Tao Kong, Lingpeng |
| contents | Autonomous interaction with the computer has been a longstanding challenge with great potential, and the recent proliferation of large language models (LLMs) has markedly accelerated progress in building digital agents. However, most of these agents are designed to interact with a narrow domain, such as a specific software or website. This narrow focus constrains their applicability for general computer tasks. To this end, we introduce OS-Copilot, a framework to build generalist agents capable of interfacing with comprehensive elements in an operating system (OS), including the web, code terminals, files, multimedia, and various third-party applications. We use OS-Copilot to create FRIDAY, a self-improving embodied agent for automating general computer tasks. On GAIA, a general AI assistants benchmark, FRIDAY outperforms previous methods by 35%, showcasing strong generalization to unseen applications via accumulated skills from previous tasks. We also present numerical and quantitative evidence that FRIDAY learns to control and self-improve on Excel and Powerpoint with minimal supervision. Our OS-Copilot framework and empirical findings provide infrastructure and insights for future research toward more capable and general-purpose computer agents. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_07456 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | OS-Copilot: Towards Generalist Computer Agents with Self-Improvement Wu, Zhiyong Han, Chengcheng Ding, Zichen Weng, Zhenmin Liu, Zhoumianze Yao, Shunyu Yu, Tao Kong, Lingpeng Artificial Intelligence Autonomous interaction with the computer has been a longstanding challenge with great potential, and the recent proliferation of large language models (LLMs) has markedly accelerated progress in building digital agents. However, most of these agents are designed to interact with a narrow domain, such as a specific software or website. This narrow focus constrains their applicability for general computer tasks. To this end, we introduce OS-Copilot, a framework to build generalist agents capable of interfacing with comprehensive elements in an operating system (OS), including the web, code terminals, files, multimedia, and various third-party applications. We use OS-Copilot to create FRIDAY, a self-improving embodied agent for automating general computer tasks. On GAIA, a general AI assistants benchmark, FRIDAY outperforms previous methods by 35%, showcasing strong generalization to unseen applications via accumulated skills from previous tasks. We also present numerical and quantitative evidence that FRIDAY learns to control and self-improve on Excel and Powerpoint with minimal supervision. Our OS-Copilot framework and empirical findings provide infrastructure and insights for future research toward more capable and general-purpose computer agents. |
| title | OS-Copilot: Towards Generalist Computer Agents with Self-Improvement |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2402.07456 |