GUICourse: From General Vision Language Models to Versatile GUI Agents
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866908384479936512 |
|---|---|
| author | Chen, Wentong Cui, Junbo Hu, Jinyi Qin, Yujia Fang, Junjie Zhao, Yue Wang, Chongyi Liu, Jun Chen, Guirong Huo, Yupeng Yao, Yuan Lin, Yankai Liu, Zhiyuan Sun, Maosong |
| author_facet | Chen, Wentong Cui, Junbo Hu, Jinyi Qin, Yujia Fang, Junjie Zhao, Yue Wang, Chongyi Liu, Jun Chen, Guirong Huo, Yupeng Yao, Yuan Lin, Yankai Liu, Zhiyuan Sun, Maosong |
| contents | Utilizing Graphic User Interface (GUI) for human-computer interaction is essential for accessing a wide range of digital tools. Recent advancements in Vision Language Models (VLMs) highlight the compelling potential to develop versatile agents to help humans finish GUI navigation tasks. However, current VLMs are challenged in terms of fundamental abilities (OCR and grounding) and GUI knowledge (the functions and control methods of GUI elements), preventing them from becoming practical GUI agents. To solve these challenges, we contribute GUICourse, a suite of datasets to train visual-based GUI agents from general VLMs. First, we introduce the GUIEnv dataset to strengthen the OCR and grounding capabilities of VLMs. Then, we introduce the GUIAct and GUIChat datasets to enrich their knowledge of GUI components and interactions. Experiments demonstrate that our GUI agents have better performance on common GUI tasks than their baseline VLMs. Even the small-size GUI agent (with 3.1B parameters) can still work well on single-step and multi-step GUI tasks. Finally, we analyze the different varieties in the training stage of this agent by ablation study. Our source codes and datasets are released at https://github.com/yiye3/GUICourse. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_11317 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | GUICourse: From General Vision Language Models to Versatile GUI Agents Chen, Wentong Cui, Junbo Hu, Jinyi Qin, Yujia Fang, Junjie Zhao, Yue Wang, Chongyi Liu, Jun Chen, Guirong Huo, Yupeng Yao, Yuan Lin, Yankai Liu, Zhiyuan Sun, Maosong Artificial Intelligence Computation and Language Computer Vision and Pattern Recognition Human-Computer Interaction Utilizing Graphic User Interface (GUI) for human-computer interaction is essential for accessing a wide range of digital tools. Recent advancements in Vision Language Models (VLMs) highlight the compelling potential to develop versatile agents to help humans finish GUI navigation tasks. However, current VLMs are challenged in terms of fundamental abilities (OCR and grounding) and GUI knowledge (the functions and control methods of GUI elements), preventing them from becoming practical GUI agents. To solve these challenges, we contribute GUICourse, a suite of datasets to train visual-based GUI agents from general VLMs. First, we introduce the GUIEnv dataset to strengthen the OCR and grounding capabilities of VLMs. Then, we introduce the GUIAct and GUIChat datasets to enrich their knowledge of GUI components and interactions. Experiments demonstrate that our GUI agents have better performance on common GUI tasks than their baseline VLMs. Even the small-size GUI agent (with 3.1B parameters) can still work well on single-step and multi-step GUI tasks. Finally, we analyze the different varieties in the training stage of this agent by ablation study. Our source codes and datasets are released at https://github.com/yiye3/GUICourse. |
| title | GUICourse: From General Vision Language Models to Versatile GUI Agents |
| topic | Artificial Intelligence Computation and Language Computer Vision and Pattern Recognition Human-Computer Interaction |
| url | https://arxiv.org/abs/2406.11317 |