DigiData: Training and Evaluating General-Purpose Mobile Control Agents
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866911260850782208 |
|---|---|
| author | Sun, Yuxuan Wang, Manchen Qian, Shengyi Wong, William R. Gan, Eric D'Oro, Pierluca Munoz, Alejandro Castillejo Silwal, Sneha Matias, Pedro Kamra, Nitin Kottur, Satwik Raines, Nick Zhao, Xuanyi Chen, Joy Greer, Joseph Madotto, Andrea Bolourchi, Allen Valori, James Carlberg, Kevin Ridgeway, Karl Tighe, Joseph |
| author_facet | Sun, Yuxuan Wang, Manchen Qian, Shengyi Wong, William R. Gan, Eric D'Oro, Pierluca Munoz, Alejandro Castillejo Silwal, Sneha Matias, Pedro Kamra, Nitin Kottur, Satwik Raines, Nick Zhao, Xuanyi Chen, Joy Greer, Joseph Madotto, Andrea Bolourchi, Allen Valori, James Carlberg, Kevin Ridgeway, Karl Tighe, Joseph |
| contents | AI agents capable of controlling user interfaces have the potential to transform human interaction with digital devices. To accelerate this transformation, two fundamental building blocks are essential: high-quality datasets that enable agents to achieve complex and human-relevant goals, and robust evaluation methods that allow researchers and practitioners to rapidly enhance agent performance. In this paper, we introduce DigiData, a large-scale, high-quality, diverse, multi-modal dataset designed for training mobile control agents. Unlike existing datasets, which derive goals from unstructured interactions, DigiData is meticulously constructed through comprehensive exploration of app features, resulting in greater diversity and higher goal complexity. Additionally, we present DigiData-Bench, a benchmark for evaluating mobile control agents on real-world complex tasks. We demonstrate that the commonly used step-accuracy metric falls short in reliably assessing mobile control agents and, to address this, we propose dynamic evaluation protocols and AI-powered evaluations as rigorous alternatives for agent assessment. Our contributions aim to significantly advance the development of mobile control agents, paving the way for more intuitive and effective human-device interactions. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_07413 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | DigiData: Training and Evaluating General-Purpose Mobile Control Agents Sun, Yuxuan Wang, Manchen Qian, Shengyi Wong, William R. Gan, Eric D'Oro, Pierluca Munoz, Alejandro Castillejo Silwal, Sneha Matias, Pedro Kamra, Nitin Kottur, Satwik Raines, Nick Zhao, Xuanyi Chen, Joy Greer, Joseph Madotto, Andrea Bolourchi, Allen Valori, James Carlberg, Kevin Ridgeway, Karl Tighe, Joseph Artificial Intelligence Computation and Language Human-Computer Interaction Machine Learning AI agents capable of controlling user interfaces have the potential to transform human interaction with digital devices. To accelerate this transformation, two fundamental building blocks are essential: high-quality datasets that enable agents to achieve complex and human-relevant goals, and robust evaluation methods that allow researchers and practitioners to rapidly enhance agent performance. In this paper, we introduce DigiData, a large-scale, high-quality, diverse, multi-modal dataset designed for training mobile control agents. Unlike existing datasets, which derive goals from unstructured interactions, DigiData is meticulously constructed through comprehensive exploration of app features, resulting in greater diversity and higher goal complexity. Additionally, we present DigiData-Bench, a benchmark for evaluating mobile control agents on real-world complex tasks. We demonstrate that the commonly used step-accuracy metric falls short in reliably assessing mobile control agents and, to address this, we propose dynamic evaluation protocols and AI-powered evaluations as rigorous alternatives for agent assessment. Our contributions aim to significantly advance the development of mobile control agents, paving the way for more intuitive and effective human-device interactions. |
| title | DigiData: Training and Evaluating General-Purpose Mobile Control Agents |
| topic | Artificial Intelligence Computation and Language Human-Computer Interaction Machine Learning |
| url | https://arxiv.org/abs/2511.07413 |