How Auxiliary Reasoning Unleashes GUI Grounding in VLMs
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908538937278464 |
|---|---|
| author | Li, Weiming Shao, Yan Yang, Jing Lu, Yujing Zhong, Ling Wang, Yuhan Duan, Manni |
| author_facet | Li, Weiming Shao, Yan Yang, Jing Lu, Yujing Zhong, Ling Wang, Yuhan Duan, Manni |
| contents | Graphical user interface (GUI) grounding is a fundamental task for building GUI agents. However, general vision-language models (VLMs) struggle with this task due to a lack of specific optimization. We identify a key gap in this paper: while VLMs exhibit significant latent grounding potential, as demonstrated by their performance measured by Pointing Game, they underperform when tasked with outputting explicit coordinates. To address this discrepancy, and bypass the high data and annotation costs of current fine-tuning approaches, we propose three zero-shot auxiliary reasoning methods. By providing explicit spatial cues such as axes, grids and labeled intersections as part of the input image, these methods enable VLMs to articulate their implicit spatial understanding capabilities. We evaluate these methods on four GUI grounding benchmarks across seven open-source and proprietary VLMs. The evaluation results demonstrate that the proposed methods substantially improve the performance of GUI grounding. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_11548 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | How Auxiliary Reasoning Unleashes GUI Grounding in VLMs Li, Weiming Shao, Yan Yang, Jing Lu, Yujing Zhong, Ling Wang, Yuhan Duan, Manni Computer Vision and Pattern Recognition Graphical user interface (GUI) grounding is a fundamental task for building GUI agents. However, general vision-language models (VLMs) struggle with this task due to a lack of specific optimization. We identify a key gap in this paper: while VLMs exhibit significant latent grounding potential, as demonstrated by their performance measured by Pointing Game, they underperform when tasked with outputting explicit coordinates. To address this discrepancy, and bypass the high data and annotation costs of current fine-tuning approaches, we propose three zero-shot auxiliary reasoning methods. By providing explicit spatial cues such as axes, grids and labeled intersections as part of the input image, these methods enable VLMs to articulate their implicit spatial understanding capabilities. We evaluate these methods on four GUI grounding benchmarks across seven open-source and proprietary VLMs. The evaluation results demonstrate that the proposed methods substantially improve the performance of GUI grounding. |
| title | How Auxiliary Reasoning Unleashes GUI Grounding in VLMs |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2509.11548 |