AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866913779140263936 |
|---|---|
| author | Rawles, Christopher Clinckemaillie, Sarah Chang, Yifan Waltz, Jonathan Lau, Gabrielle Fair, Marybeth Li, Alice Bishop, William Li, Wei Campbell-Ajala, Folawiyo Toyama, Daniel Berry, Robert Tyamagundlu, Divya Lillicrap, Timothy Riva, Oriana |
| author_facet | Rawles, Christopher Clinckemaillie, Sarah Chang, Yifan Waltz, Jonathan Lau, Gabrielle Fair, Marybeth Li, Alice Bishop, William Li, Wei Campbell-Ajala, Folawiyo Toyama, Daniel Berry, Robert Tyamagundlu, Divya Lillicrap, Timothy Riva, Oriana |
| contents | Autonomous agents that execute human tasks by controlling computers can enhance human productivity and application accessibility. However, progress in this field will be driven by realistic and reproducible benchmarks. We present AndroidWorld, a fully functional Android environment that provides reward signals for 116 programmatic tasks across 20 real-world Android apps. Unlike existing interactive environments, which provide a static test set, AndroidWorld dynamically constructs tasks that are parameterized and expressed in natural language in unlimited ways, thus enabling testing on a much larger and more realistic suite of tasks. To ensure reproducibility, each task includes dedicated initialization, success-checking, and tear-down logic, which modifies and inspects the device's system state. We experiment with baseline agents to test AndroidWorld and provide initial results on the benchmark. Our best agent can complete 30.6% of AndroidWorld's tasks, leaving ample room for future work. Furthermore, we adapt a popular desktop web agent to work on Android, which we find to be less effective on mobile, suggesting future research is needed to achieve universal, cross-platform agents. Finally, we also conduct a robustness analysis, showing that task variations can significantly affect agent performance, demonstrating that without such testing, agent performance metrics may not fully reflect practical challenges. AndroidWorld and the experiments in this paper are available at github.com/google-research/android_world. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_14573 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents Rawles, Christopher Clinckemaillie, Sarah Chang, Yifan Waltz, Jonathan Lau, Gabrielle Fair, Marybeth Li, Alice Bishop, William Li, Wei Campbell-Ajala, Folawiyo Toyama, Daniel Berry, Robert Tyamagundlu, Divya Lillicrap, Timothy Riva, Oriana Artificial Intelligence Machine Learning Autonomous agents that execute human tasks by controlling computers can enhance human productivity and application accessibility. However, progress in this field will be driven by realistic and reproducible benchmarks. We present AndroidWorld, a fully functional Android environment that provides reward signals for 116 programmatic tasks across 20 real-world Android apps. Unlike existing interactive environments, which provide a static test set, AndroidWorld dynamically constructs tasks that are parameterized and expressed in natural language in unlimited ways, thus enabling testing on a much larger and more realistic suite of tasks. To ensure reproducibility, each task includes dedicated initialization, success-checking, and tear-down logic, which modifies and inspects the device's system state. We experiment with baseline agents to test AndroidWorld and provide initial results on the benchmark. Our best agent can complete 30.6% of AndroidWorld's tasks, leaving ample room for future work. Furthermore, we adapt a popular desktop web agent to work on Android, which we find to be less effective on mobile, suggesting future research is needed to achieve universal, cross-platform agents. Finally, we also conduct a robustness analysis, showing that task variations can significantly affect agent performance, demonstrating that without such testing, agent performance metrics may not fully reflect practical challenges. AndroidWorld and the experiments in this paper are available at github.com/google-research/android_world. |
| title | AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents |
| topic | Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2405.14573 |