E-ANT: A Large-Scale Dataset for Efficient Automatic GUI NavigaTion
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866929403450097664 |
|---|---|
| author | Wang, Ke Xia, Tianyu Gu, Zhangxuan Zhao, Yi Shen, Shuheng Meng, Changhua Wang, Weiqiang Xu, Ke |
| author_facet | Wang, Ke Xia, Tianyu Gu, Zhangxuan Zhao, Yi Shen, Shuheng Meng, Changhua Wang, Weiqiang Xu, Ke |
| contents | Online GUI navigation on mobile devices has driven a lot of attention recent years since it contributes to many real-world applications. With the rapid development of large language models (LLM), multimodal large language models (MLLM) have tremendous potential on this task. However, existing MLLMs need high quality data to improve its abilities of making the correct navigation decisions according to the human user inputs. In this paper, we developed a novel and highly valuable dataset, named \textbf{E-ANT}, as the first Chinese GUI navigation dataset that contains real human behaviour and high quality screenshots with annotations, containing nearly 40,000 real human traces over 5000+ different tinyAPPs. Furthermore, we evaluate various powerful MLLMs on E-ANT and show their experiments results with sufficient ablations. We believe that our proposed dataset will be beneficial for both the evaluation and development of GUI navigation and LLM/MLLM decision-making capabilities. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_14250 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | E-ANT: A Large-Scale Dataset for Efficient Automatic GUI NavigaTion Wang, Ke Xia, Tianyu Gu, Zhangxuan Zhao, Yi Shen, Shuheng Meng, Changhua Wang, Weiqiang Xu, Ke Computer Vision and Pattern Recognition Human-Computer Interaction Online GUI navigation on mobile devices has driven a lot of attention recent years since it contributes to many real-world applications. With the rapid development of large language models (LLM), multimodal large language models (MLLM) have tremendous potential on this task. However, existing MLLMs need high quality data to improve its abilities of making the correct navigation decisions according to the human user inputs. In this paper, we developed a novel and highly valuable dataset, named \textbf{E-ANT}, as the first Chinese GUI navigation dataset that contains real human behaviour and high quality screenshots with annotations, containing nearly 40,000 real human traces over 5000+ different tinyAPPs. Furthermore, we evaluate various powerful MLLMs on E-ANT and show their experiments results with sufficient ablations. We believe that our proposed dataset will be beneficial for both the evaluation and development of GUI navigation and LLM/MLLM decision-making capabilities. |
| title | E-ANT: A Large-Scale Dataset for Efficient Automatic GUI NavigaTion |
| topic | Computer Vision and Pattern Recognition Human-Computer Interaction |
| url | https://arxiv.org/abs/2406.14250 |