Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2412.13501 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915516747087872 |
|---|---|
| author | Nguyen, Dang Chen, Jian Wang, Yu Wu, Gang Park, Namyong Hu, Zhengmian Lyu, Hanjia Wu, Junda Aponte, Ryan Xia, Yu Li, Xintong Shi, Jing Chen, Hongjie Lai, Viet Dac Xie, Zhouhang Kim, Sungchul Zhang, Ruiyi Yu, Tong Tanjim, Mehrab Ahmed, Nesreen K. Mathur, Puneet Yoon, Seunghyun Yao, Lina Kveton, Branislav Kil, Jihyung Nguyen, Thien Huu Bui, Trung Zhou, Tianyi Rossi, Ryan A. Dernoncourt, Franck |
| author_facet | Nguyen, Dang Chen, Jian Wang, Yu Wu, Gang Park, Namyong Hu, Zhengmian Lyu, Hanjia Wu, Junda Aponte, Ryan Xia, Yu Li, Xintong Shi, Jing Chen, Hongjie Lai, Viet Dac Xie, Zhouhang Kim, Sungchul Zhang, Ruiyi Yu, Tong Tanjim, Mehrab Ahmed, Nesreen K. Mathur, Puneet Yoon, Seunghyun Yao, Lina Kveton, Branislav Kil, Jihyung Nguyen, Thien Huu Bui, Trung Zhou, Tianyi Rossi, Ryan A. Dernoncourt, Franck |
| contents | Graphical User Interface (GUI) agents, powered by Large Foundation Models, have emerged as a transformative approach to automating human-computer interaction. These agents autonomously interact with digital systems or software applications via GUIs, emulating human actions such as clicking, typing, and navigating visual elements across diverse platforms. Motivated by the growing interest and fundamental importance of GUI agents, we provide a comprehensive survey that categorizes their benchmarks, evaluation metrics, architectures, and training methods. We propose a unified framework that delineates their perception, reasoning, planning, and acting capabilities. Furthermore, we identify important open challenges and discuss key future directions. Finally, this work serves as a basis for practitioners and researchers to gain an intuitive understanding of current progress, techniques, benchmarks, and critical open problems that remain to be addressed. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_13501 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | GUI Agents: A Survey Nguyen, Dang Chen, Jian Wang, Yu Wu, Gang Park, Namyong Hu, Zhengmian Lyu, Hanjia Wu, Junda Aponte, Ryan Xia, Yu Li, Xintong Shi, Jing Chen, Hongjie Lai, Viet Dac Xie, Zhouhang Kim, Sungchul Zhang, Ruiyi Yu, Tong Tanjim, Mehrab Ahmed, Nesreen K. Mathur, Puneet Yoon, Seunghyun Yao, Lina Kveton, Branislav Kil, Jihyung Nguyen, Thien Huu Bui, Trung Zhou, Tianyi Rossi, Ryan A. Dernoncourt, Franck Artificial Intelligence Human-Computer Interaction Graphical User Interface (GUI) agents, powered by Large Foundation Models, have emerged as a transformative approach to automating human-computer interaction. These agents autonomously interact with digital systems or software applications via GUIs, emulating human actions such as clicking, typing, and navigating visual elements across diverse platforms. Motivated by the growing interest and fundamental importance of GUI agents, we provide a comprehensive survey that categorizes their benchmarks, evaluation metrics, architectures, and training methods. We propose a unified framework that delineates their perception, reasoning, planning, and acting capabilities. Furthermore, we identify important open challenges and discuss key future directions. Finally, this work serves as a basis for practitioners and researchers to gain an intuitive understanding of current progress, techniques, benchmarks, and critical open problems that remain to be addressed. |
| title | GUI Agents: A Survey |
| topic | Artificial Intelligence Human-Computer Interaction |
| url | https://arxiv.org/abs/2412.13501 |