A Survey on (M)LLM-Based GUI Agents
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866910985544007680 |
|---|---|
| author | Tang, Fei Xu, Haolei Zhang, Hang Chen, Siqi Wu, Xingyu Shen, Yongliang Zhang, Wenqi Hou, Guiyang Tan, Zeqi Yan, Yuchen Song, Kaitao Shao, Jian Lu, Weiming Xiao, Jun Zhuang, Yueting |
| author_facet | Tang, Fei Xu, Haolei Zhang, Hang Chen, Siqi Wu, Xingyu Shen, Yongliang Zhang, Wenqi Hou, Guiyang Tan, Zeqi Yan, Yuchen Song, Kaitao Shao, Jian Lu, Weiming Xiao, Jun Zhuang, Yueting |
| contents | Graphical User Interface (GUI) Agents have emerged as a transformative paradigm in human-computer interaction, evolving from rule-based automation scripts to sophisticated AI-driven systems capable of understanding and executing complex interface operations. This survey provides a comprehensive examination of the rapidly advancing field of LLM-based GUI Agents, systematically analyzing their architectural foundations, technical components, and evaluation methodologies. We identify and analyze four fundamental components that constitute modern GUI Agents: (1) perception systems that integrate text-based parsing with multimodal understanding for comprehensive interface comprehension; (2) exploration mechanisms that construct and maintain knowledge bases through internal modeling, historical experience, and external information retrieval; (3) planning frameworks that leverage advanced reasoning methodologies for task decomposition and execution; and (4) interaction systems that manage action generation with robust safety controls. Through rigorous analysis of these components, we reveal how recent advances in large language models and multimodal learning have revolutionized GUI automation across desktop, mobile, and web platforms. We critically examine current evaluation frameworks, highlighting methodological limitations in existing benchmarks while proposing directions for standardization. This survey also identifies key technical challenges, including accurate element localization, effective knowledge retrieval, long-horizon planning, and safety-aware execution control, while outlining promising research directions for enhancing GUI Agents' capabilities. Our systematic review provides researchers and practitioners with a thorough understanding of the field's current state and offers insights into future developments in intelligent interface automation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_13865 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | A Survey on (M)LLM-Based GUI Agents Tang, Fei Xu, Haolei Zhang, Hang Chen, Siqi Wu, Xingyu Shen, Yongliang Zhang, Wenqi Hou, Guiyang Tan, Zeqi Yan, Yuchen Song, Kaitao Shao, Jian Lu, Weiming Xiao, Jun Zhuang, Yueting Human-Computer Interaction Artificial Intelligence Computation and Language Computer Vision and Pattern Recognition Graphical User Interface (GUI) Agents have emerged as a transformative paradigm in human-computer interaction, evolving from rule-based automation scripts to sophisticated AI-driven systems capable of understanding and executing complex interface operations. This survey provides a comprehensive examination of the rapidly advancing field of LLM-based GUI Agents, systematically analyzing their architectural foundations, technical components, and evaluation methodologies. We identify and analyze four fundamental components that constitute modern GUI Agents: (1) perception systems that integrate text-based parsing with multimodal understanding for comprehensive interface comprehension; (2) exploration mechanisms that construct and maintain knowledge bases through internal modeling, historical experience, and external information retrieval; (3) planning frameworks that leverage advanced reasoning methodologies for task decomposition and execution; and (4) interaction systems that manage action generation with robust safety controls. Through rigorous analysis of these components, we reveal how recent advances in large language models and multimodal learning have revolutionized GUI automation across desktop, mobile, and web platforms. We critically examine current evaluation frameworks, highlighting methodological limitations in existing benchmarks while proposing directions for standardization. This survey also identifies key technical challenges, including accurate element localization, effective knowledge retrieval, long-horizon planning, and safety-aware execution control, while outlining promising research directions for enhancing GUI Agents' capabilities. Our systematic review provides researchers and practitioners with a thorough understanding of the field's current state and offers insights into future developments in intelligent interface automation. |
| title | A Survey on (M)LLM-Based GUI Agents |
| topic | Human-Computer Interaction Artificial Intelligence Computation and Language Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2504.13865 |