GUI Agents with Foundation Models: A Comprehensive Survey
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866916611330408448 |
|---|---|
| author | Wang, Shuai Liu, Weiwen Chen, Jingxuan Zhou, Yuqi Gan, Weinan Zeng, Xingshan Che, Yuhan Yu, Shuai Hao, Xinlong Shao, Kun Wang, Bin Wu, Chuhan Wang, Yasheng Tang, Ruiming Hao, Jianye |
| author_facet | Wang, Shuai Liu, Weiwen Chen, Jingxuan Zhou, Yuqi Gan, Weinan Zeng, Xingshan Che, Yuhan Yu, Shuai Hao, Xinlong Shao, Kun Wang, Bin Wu, Chuhan Wang, Yasheng Tang, Ruiming Hao, Jianye |
| contents | Recent advances in foundation models, particularly Large Language Models (LLMs) and Multimodal Large Language Models (MLLMs), have facilitated the development of intelligent agents capable of performing complex tasks. By leveraging the ability of (M)LLMs to process and interpret Graphical User Interfaces (GUIs), these agents can autonomously execute user instructions, simulating human-like interactions such as clicking and typing. This survey consolidates recent research on (M)LLM-based GUI agents, highlighting key innovations in data resources, frameworks, and applications. We begin by reviewing representative datasets and benchmarks, followed by an overview of a generalized, unified framework that encapsulates the essential components of prior studies, supported by a detailed taxonomy. Additionally, we explore relevant commercial applications. Drawing insights from existing work, we identify key challenges and propose future research directions. We hope this survey will inspire further advancements in the field of (M)LLM-based GUI agents. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_04890 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | GUI Agents with Foundation Models: A Comprehensive Survey Wang, Shuai Liu, Weiwen Chen, Jingxuan Zhou, Yuqi Gan, Weinan Zeng, Xingshan Che, Yuhan Yu, Shuai Hao, Xinlong Shao, Kun Wang, Bin Wu, Chuhan Wang, Yasheng Tang, Ruiming Hao, Jianye Artificial Intelligence Human-Computer Interaction Recent advances in foundation models, particularly Large Language Models (LLMs) and Multimodal Large Language Models (MLLMs), have facilitated the development of intelligent agents capable of performing complex tasks. By leveraging the ability of (M)LLMs to process and interpret Graphical User Interfaces (GUIs), these agents can autonomously execute user instructions, simulating human-like interactions such as clicking and typing. This survey consolidates recent research on (M)LLM-based GUI agents, highlighting key innovations in data resources, frameworks, and applications. We begin by reviewing representative datasets and benchmarks, followed by an overview of a generalized, unified framework that encapsulates the essential components of prior studies, supported by a detailed taxonomy. Additionally, we explore relevant commercial applications. Drawing insights from existing work, we identify key challenges and propose future research directions. We hope this survey will inspire further advancements in the field of (M)LLM-based GUI agents. |
| title | GUI Agents with Foundation Models: A Comprehensive Survey |
| topic | Artificial Intelligence Human-Computer Interaction |
| url | https://arxiv.org/abs/2411.04890 |