Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2505.12370 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912391647723520 |
|---|---|
| author | Yuan, Xinbin Zhang, Jian Li, Kaixin Cai, Zhuoxuan Yao, Lujian Chen, Jie Wang, Enguang Hou, Qibin Chen, Jinwei Jiang, Peng-Tao Li, Bo |
| author_facet | Yuan, Xinbin Zhang, Jian Li, Kaixin Cai, Zhuoxuan Yao, Lujian Chen, Jie Wang, Enguang Hou, Qibin Chen, Jinwei Jiang, Peng-Tao Li, Bo |
| contents | Graphical User Interface (GUI) agents have made substantial strides in understanding and executing user instructions across diverse platforms. Yet, grounding these instructions to precise interface elements remains challenging, especially in complex, high-resolution, professional environments. Traditional supervised finetuning (SFT) methods often require large volumes of diverse data and exhibit weak generalization. To overcome these limitations, we introduce a reinforcement learning (RL) based framework that incorporates three core strategies: (1) seed data curation to ensure high quality training samples, (2) a dense policy gradient that provides continuous feedback based on prediction accuracy, and (3) a self evolutionary reinforcement finetuning mechanism that iteratively refines the model using attention maps. With only 3k training samples, our 7B-parameter model achieves state-of-the-art results among similarly sized models on three grounding benchmarks. Notably, it attains 47.3\% accuracy on the ScreenSpot-Pro dataset, outperforming much larger models, such as UI-TARS-72B, by a margin of 24.2\%. These findings underscore the effectiveness of RL-based approaches in enhancing GUI agent performance, particularly in high-resolution, complex environments. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_12370 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Enhancing Visual Grounding for GUI Agents via Self-Evolutionary Reinforcement Learning Yuan, Xinbin Zhang, Jian Li, Kaixin Cai, Zhuoxuan Yao, Lujian Chen, Jie Wang, Enguang Hou, Qibin Chen, Jinwei Jiang, Peng-Tao Li, Bo Artificial Intelligence Graphical User Interface (GUI) agents have made substantial strides in understanding and executing user instructions across diverse platforms. Yet, grounding these instructions to precise interface elements remains challenging, especially in complex, high-resolution, professional environments. Traditional supervised finetuning (SFT) methods often require large volumes of diverse data and exhibit weak generalization. To overcome these limitations, we introduce a reinforcement learning (RL) based framework that incorporates three core strategies: (1) seed data curation to ensure high quality training samples, (2) a dense policy gradient that provides continuous feedback based on prediction accuracy, and (3) a self evolutionary reinforcement finetuning mechanism that iteratively refines the model using attention maps. With only 3k training samples, our 7B-parameter model achieves state-of-the-art results among similarly sized models on three grounding benchmarks. Notably, it attains 47.3\% accuracy on the ScreenSpot-Pro dataset, outperforming much larger models, such as UI-TARS-72B, by a margin of 24.2\%. These findings underscore the effectiveness of RL-based approaches in enhancing GUI agent performance, particularly in high-resolution, complex environments. |
| title | Enhancing Visual Grounding for GUI Agents via Self-Evolutionary Reinforcement Learning |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2505.12370 |