AMEX: Android Multi-annotation Expo Dataset for Mobile GUI Agents
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866916897026473984 |
|---|---|
| author | Chai, Yuxiang Huang, Siyuan Niu, Yazhe Xiao, Han Liu, Liang Zhang, Dingyu Ren, Shuai Li, Hongsheng |
| author_facet | Chai, Yuxiang Huang, Siyuan Niu, Yazhe Xiao, Han Liu, Liang Zhang, Dingyu Ren, Shuai Li, Hongsheng |
| contents | AI agents have drawn increasing attention mostly on their ability to perceive environments, understand tasks, and autonomously achieve goals. To advance research on AI agents in mobile scenarios, we introduce the Android Multi-annotation EXpo (AMEX), a comprehensive, large-scale dataset designed for generalist mobile GUI-control agents which are capable of completing tasks by directly interacting with the graphical user interface (GUI) on mobile devices. AMEX comprises over 104K high-resolution screenshots from popular mobile applications, which are annotated at multiple levels. Unlike existing GUI-related datasets, e.g., Rico, AitW, etc., AMEX includes three levels of annotations: GUI interactive element grounding, GUI screen and element functionality descriptions, and complex natural language instructions with stepwise GUI-action chains. We develop this dataset from a more instructive and detailed perspective, complementing the general settings of existing datasets. Additionally, we finetune a baseline model SPHINX Agent and illustrate the effectiveness of AMEX.The project is available at https://yxchai.com/AMEX/. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_17490 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | AMEX: Android Multi-annotation Expo Dataset for Mobile GUI Agents Chai, Yuxiang Huang, Siyuan Niu, Yazhe Xiao, Han Liu, Liang Zhang, Dingyu Ren, Shuai Li, Hongsheng Human-Computer Interaction Artificial Intelligence Multimedia AI agents have drawn increasing attention mostly on their ability to perceive environments, understand tasks, and autonomously achieve goals. To advance research on AI agents in mobile scenarios, we introduce the Android Multi-annotation EXpo (AMEX), a comprehensive, large-scale dataset designed for generalist mobile GUI-control agents which are capable of completing tasks by directly interacting with the graphical user interface (GUI) on mobile devices. AMEX comprises over 104K high-resolution screenshots from popular mobile applications, which are annotated at multiple levels. Unlike existing GUI-related datasets, e.g., Rico, AitW, etc., AMEX includes three levels of annotations: GUI interactive element grounding, GUI screen and element functionality descriptions, and complex natural language instructions with stepwise GUI-action chains. We develop this dataset from a more instructive and detailed perspective, complementing the general settings of existing datasets. Additionally, we finetune a baseline model SPHINX Agent and illustrate the effectiveness of AMEX.The project is available at https://yxchai.com/AMEX/. |
| title | AMEX: Android Multi-annotation Expo Dataset for Mobile GUI Agents |
| topic | Human-Computer Interaction Artificial Intelligence Multimedia |
| url | https://arxiv.org/abs/2407.17490 |