AndroidLens: Long-latency Evaluation with Nested Sub-targets for Android GUI Agents
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911337746006016 |
|---|---|
| author | Cao, Yue Wang, Yingyao Bu, Pi Xing, Jingxuan Jiang, Wei Zhu, Zekun Ma, Junpeng Zhou, Sashuai Lu, Tong Song, Jun Cheng, Yu Jiang, Yuning Zheng, Bo |
| author_facet | Cao, Yue Wang, Yingyao Bu, Pi Xing, Jingxuan Jiang, Wei Zhu, Zekun Ma, Junpeng Zhou, Sashuai Lu, Tong Song, Jun Cheng, Yu Jiang, Yuning Zheng, Bo |
| contents | Graphical user interface (GUI) agents can substantially improve productivity by automating frequently executed long-latency tasks on mobile devices. However, existing evaluation benchmarks are still constrained to limited applications, simple tasks, and coarse-grained metrics. To address this, we introduce AndroidLens, a challenging evaluation framework for mobile GUI agents, comprising 571 long-latency tasks in both Chinese and English environments, each requiring an average of more than 26 steps to complete. The framework features: (1) tasks derived from real-world user scenarios across 38 domains, covering complex types such as multi-constraint, multi-goal, and domain-specific tasks; (2) static evaluation that preserves real-world anomalies and allows multiple valid paths to reduce bias; and (3) dynamic evaluation that employs a milestone-based scheme for fine-grained progress measurement via Average Task Progress (ATP). Our evaluation indicates that even the best models reach only a 12.7% task success rate and 50.47% ATP. We also underscore key challenges in real-world environments, including environmental anomalies, adaptive exploration, and long-term memory retention. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_21302 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | AndroidLens: Long-latency Evaluation with Nested Sub-targets for Android GUI Agents Cao, Yue Wang, Yingyao Bu, Pi Xing, Jingxuan Jiang, Wei Zhu, Zekun Ma, Junpeng Zhou, Sashuai Lu, Tong Song, Jun Cheng, Yu Jiang, Yuning Zheng, Bo Computer Vision and Pattern Recognition Graphical user interface (GUI) agents can substantially improve productivity by automating frequently executed long-latency tasks on mobile devices. However, existing evaluation benchmarks are still constrained to limited applications, simple tasks, and coarse-grained metrics. To address this, we introduce AndroidLens, a challenging evaluation framework for mobile GUI agents, comprising 571 long-latency tasks in both Chinese and English environments, each requiring an average of more than 26 steps to complete. The framework features: (1) tasks derived from real-world user scenarios across 38 domains, covering complex types such as multi-constraint, multi-goal, and domain-specific tasks; (2) static evaluation that preserves real-world anomalies and allows multiple valid paths to reduce bias; and (3) dynamic evaluation that employs a milestone-based scheme for fine-grained progress measurement via Average Task Progress (ATP). Our evaluation indicates that even the best models reach only a 12.7% task success rate and 50.47% ATP. We also underscore key challenges in real-world environments, including environmental anomalies, adaptive exploration, and long-term memory retention. |
| title | AndroidLens: Long-latency Evaluation with Nested Sub-targets for Android GUI Agents |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2512.21302 |