Building Autonomous GUI Navigation via Agentic-Q Estimation and Step-Wise Policy Optimization
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2026
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866914329762201600 |
|---|---|
| author | Wang, Yibo Huzhang, Guangda Hu, Yuwei Xia, Yu Lu, Shiyin Chen, Qing-Guo Xu, Zhao Luo, Weihua Zhang, Kaifu Zhang, Lijun |
| author_facet | Wang, Yibo Huzhang, Guangda Hu, Yuwei Xia, Yu Lu, Shiyin Chen, Qing-Guo Xu, Zhao Luo, Weihua Zhang, Kaifu Zhang, Lijun |
| contents | Recent advances in Multimodal Large Language Models (MLLMs) have substantially driven the progress of autonomous agents for Graphical User Interface (GUI). Nevertheless, in real-world applications, GUI agents are often faced with non-stationary environments, leading to high computational costs for data curation and policy optimization. In this report, we introduce a novel MLLM-centered framework for GUI agents, which consists of two components: agentic-Q estimation and step-wise policy optimization. The former one aims to optimize a Q-model that can generate step-wise values to evaluate the contribution of a given action to task completion. The latter one takes step-wise samples from the state-action trajectory as inputs, and optimizes the policy via reinforcement learning with our agentic-Q model. It should be noticed that (i) all state-action trajectories are produced by the policy itself, so that the data collection costs are manageable; (ii) the policy update is decoupled from the environment, ensuring stable and efficient optimization. Empirical evaluations show that our framework endows Ovis2.5-9B with powerful GUI interaction capabilities, achieving remarkable performances on GUI navigation and grounding benchmarks and even surpassing contenders with larger scales. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_13653 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Building Autonomous GUI Navigation via Agentic-Q Estimation and Step-Wise Policy Optimization Wang, Yibo Huzhang, Guangda Hu, Yuwei Xia, Yu Lu, Shiyin Chen, Qing-Guo Xu, Zhao Luo, Weihua Zhang, Kaifu Zhang, Lijun Artificial Intelligence Computation and Language Computer Vision and Pattern Recognition Human-Computer Interaction Recent advances in Multimodal Large Language Models (MLLMs) have substantially driven the progress of autonomous agents for Graphical User Interface (GUI). Nevertheless, in real-world applications, GUI agents are often faced with non-stationary environments, leading to high computational costs for data curation and policy optimization. In this report, we introduce a novel MLLM-centered framework for GUI agents, which consists of two components: agentic-Q estimation and step-wise policy optimization. The former one aims to optimize a Q-model that can generate step-wise values to evaluate the contribution of a given action to task completion. The latter one takes step-wise samples from the state-action trajectory as inputs, and optimizes the policy via reinforcement learning with our agentic-Q model. It should be noticed that (i) all state-action trajectories are produced by the policy itself, so that the data collection costs are manageable; (ii) the policy update is decoupled from the environment, ensuring stable and efficient optimization. Empirical evaluations show that our framework endows Ovis2.5-9B with powerful GUI interaction capabilities, achieving remarkable performances on GUI navigation and grounding benchmarks and even surpassing contenders with larger scales. |
| title | Building Autonomous GUI Navigation via Agentic-Q Estimation and Step-Wise Policy Optimization |
| topic | Artificial Intelligence Computation and Language Computer Vision and Pattern Recognition Human-Computer Interaction |
| url | https://arxiv.org/abs/2602.13653 |