Benchmarking Mobile Device Control Agents across Diverse Configurations
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866913949006430208 |
|---|---|
| author | Lee, Juyong Min, Taywon An, Minyong Hahm, Dongyoon Lee, Haeone Kim, Changyeon Lee, Kimin |
| author_facet | Lee, Juyong Min, Taywon An, Minyong Hahm, Dongyoon Lee, Haeone Kim, Changyeon Lee, Kimin |
| contents | Mobile device control agents can largely enhance user interactions and productivity by automating daily tasks. However, despite growing interest in developing practical agents, the absence of a commonly adopted benchmark in this area makes it challenging to quantify scientific progress. In this work, we introduce B-MoCA: a novel benchmark with interactive environments for evaluating and developing mobile device control agents. To create a realistic benchmark, we develop B-MoCA based on the Android operating system and define 131 common daily tasks. Importantly, we incorporate a randomization feature that changes the configurations of mobile devices, including user interface layouts and language settings, to assess generalization performance. We benchmark diverse agents, including agents employing large language models (LLMs) or multi-modal LLMs as well as agents trained with imitation learning using human expert demonstrations. While these agents demonstrate proficiency in executing straightforward tasks, their poor performance on complex tasks highlights significant opportunities for future research to improve effectiveness. Our source code is publicly available at https://b-moca.github.io. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_16660 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Benchmarking Mobile Device Control Agents across Diverse Configurations Lee, Juyong Min, Taywon An, Minyong Hahm, Dongyoon Lee, Haeone Kim, Changyeon Lee, Kimin Human-Computer Interaction Artificial Intelligence Machine Learning Mobile device control agents can largely enhance user interactions and productivity by automating daily tasks. However, despite growing interest in developing practical agents, the absence of a commonly adopted benchmark in this area makes it challenging to quantify scientific progress. In this work, we introduce B-MoCA: a novel benchmark with interactive environments for evaluating and developing mobile device control agents. To create a realistic benchmark, we develop B-MoCA based on the Android operating system and define 131 common daily tasks. Importantly, we incorporate a randomization feature that changes the configurations of mobile devices, including user interface layouts and language settings, to assess generalization performance. We benchmark diverse agents, including agents employing large language models (LLMs) or multi-modal LLMs as well as agents trained with imitation learning using human expert demonstrations. While these agents demonstrate proficiency in executing straightforward tasks, their poor performance on complex tasks highlights significant opportunities for future research to improve effectiveness. Our source code is publicly available at https://b-moca.github.io. |
| title | Benchmarking Mobile Device Control Agents across Diverse Configurations |
| topic | Human-Computer Interaction Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2404.16660 |