Saved in:
| Main Authors: | , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2510.14388 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911425356627968 |
|---|---|
| author | Wu, Zhe Lu, Hongjin Xing, Junliang Zhang, Changhao Li, Yuxuan Zhu, Yin Yang, Yuhao Jing, Yuheng Li, Kai Shao, Kun Hao, Jianye Wang, Jun Shi, Yuanchun |
| author_facet | Wu, Zhe Lu, Hongjin Xing, Junliang Zhang, Changhao Li, Yuxuan Zhu, Yin Yang, Yuhao Jing, Yuheng Li, Kai Shao, Kun Hao, Jianye Wang, Jun Shi, Yuanchun |
| contents | Building agents that autonomously operate mobile devices has attracted increasing attention. While Vision-Language Models (VLMs) show promise, most existing approaches rely on direct state-to-action mappings, which lack structured reasoning and planning, and thus generalize poorly to novel tasks or unseen UI layouts. We introduce Hi-Agent, a trainable hierarchical vision-language agent for mobile control, featuring a high-level reasoning model and a low-level action model that are jointly optimized. For efficient training, we reformulate multi-step decision-making as a sequence of single-step subgoals and propose a foresight advantage function, which leverages execution feedback from the low-level model to guide high-level optimization. This design alleviates the path explosion issue encountered by Group Relative Policy Optimization (GRPO) in long-horizon tasks and enables stable, critic-free joint training. Hi-Agent achieves a new State-Of-The-Art (SOTA) 87.9% task success rate on the Android-in-the-Wild (AitW) benchmark, significantly outperforming prior methods across three paradigms: prompt-based (AppAgent: 17.7%), supervised (Filtered BC: 54.5%), and reinforcement learning-based (DigiRL: 71.9%). It also demonstrates competitive zero-shot generalization on the ScreenSpot-v2 benchmark. On the more challenging AndroidWorld benchmark, Hi-Agent also scales effectively with larger backbones, showing strong adaptability in high-complexity mobile control scenarios. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_14388 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Hi-Agent: Hierarchical Vision-Language Agents for Mobile Device Control Wu, Zhe Lu, Hongjin Xing, Junliang Zhang, Changhao Li, Yuxuan Zhu, Yin Yang, Yuhao Jing, Yuheng Li, Kai Shao, Kun Hao, Jianye Wang, Jun Shi, Yuanchun Artificial Intelligence Building agents that autonomously operate mobile devices has attracted increasing attention. While Vision-Language Models (VLMs) show promise, most existing approaches rely on direct state-to-action mappings, which lack structured reasoning and planning, and thus generalize poorly to novel tasks or unseen UI layouts. We introduce Hi-Agent, a trainable hierarchical vision-language agent for mobile control, featuring a high-level reasoning model and a low-level action model that are jointly optimized. For efficient training, we reformulate multi-step decision-making as a sequence of single-step subgoals and propose a foresight advantage function, which leverages execution feedback from the low-level model to guide high-level optimization. This design alleviates the path explosion issue encountered by Group Relative Policy Optimization (GRPO) in long-horizon tasks and enables stable, critic-free joint training. Hi-Agent achieves a new State-Of-The-Art (SOTA) 87.9% task success rate on the Android-in-the-Wild (AitW) benchmark, significantly outperforming prior methods across three paradigms: prompt-based (AppAgent: 17.7%), supervised (Filtered BC: 54.5%), and reinforcement learning-based (DigiRL: 71.9%). It also demonstrates competitive zero-shot generalization on the ScreenSpot-v2 benchmark. On the more challenging AndroidWorld benchmark, Hi-Agent also scales effectively with larger backbones, showing strong adaptability in high-complexity mobile control scenarios. |
| title | Hi-Agent: Hierarchical Vision-Language Agents for Mobile Device Control |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2510.14388 |