AndroidLab: Training and Systematic Benchmarking of Android Autonomous Agents
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929573817483264 |
|---|---|
| author | Xu, Yifan Liu, Xiao Sun, Xueqiao Cheng, Siyi Yu, Hao Lai, Hanyu Zhang, Shudan Zhang, Dan Tang, Jie Dong, Yuxiao |
| author_facet | Xu, Yifan Liu, Xiao Sun, Xueqiao Cheng, Siyi Yu, Hao Lai, Hanyu Zhang, Shudan Zhang, Dan Tang, Jie Dong, Yuxiao |
| contents | Autonomous agents have become increasingly important for interacting with the real world. Android agents, in particular, have been recently a frequently-mentioned interaction method. However, existing studies for training and evaluating Android agents lack systematic research on both open-source and closed-source models. In this work, we propose AndroidLab as a systematic Android agent framework. It includes an operation environment with different modalities, action space, and a reproducible benchmark. It supports both large language models (LLMs) and multimodal models (LMMs) in the same action space. AndroidLab benchmark includes predefined Android virtual devices and 138 tasks across nine apps built on these devices. By using the AndroidLab environment, we develop an Android Instruction dataset and train six open-source LLMs and LMMs, lifting the average success rates from 4.59% to 21.50% for LLMs and from 1.93% to 13.28% for LMMs. AndroidLab is open-sourced and publicly available at https://github.com/THUDM/Android-Lab. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_24024 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | AndroidLab: Training and Systematic Benchmarking of Android Autonomous Agents Xu, Yifan Liu, Xiao Sun, Xueqiao Cheng, Siyi Yu, Hao Lai, Hanyu Zhang, Shudan Zhang, Dan Tang, Jie Dong, Yuxiao Artificial Intelligence Autonomous agents have become increasingly important for interacting with the real world. Android agents, in particular, have been recently a frequently-mentioned interaction method. However, existing studies for training and evaluating Android agents lack systematic research on both open-source and closed-source models. In this work, we propose AndroidLab as a systematic Android agent framework. It includes an operation environment with different modalities, action space, and a reproducible benchmark. It supports both large language models (LLMs) and multimodal models (LMMs) in the same action space. AndroidLab benchmark includes predefined Android virtual devices and 138 tasks across nine apps built on these devices. By using the AndroidLab environment, we develop an Android Instruction dataset and train six open-source LLMs and LMMs, lifting the average success rates from 4.59% to 21.50% for LLMs and from 1.93% to 13.28% for LMMs. AndroidLab is open-sourced and publicly available at https://github.com/THUDM/Android-Lab. |
| title | AndroidLab: Training and Systematic Benchmarking of Android Autonomous Agents |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2410.24024 |