AdaZoom-GUI: Adaptive Zoom-based GUI Grounding with Instruction Refinement
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2026
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866917351377600512 |
|---|---|
| author | Pei, Siqi Tang, Liang Duan, Tiaonan Chen, Long Li, Shuxian Huang, Kaer Jing, Yanzhe Yan, Yiqiang Zhang, Bo Jiang, Chenghao Zhang, Borui Lu, Jiwen |
| author_facet | Pei, Siqi Tang, Liang Duan, Tiaonan Chen, Long Li, Shuxian Huang, Kaer Jing, Yanzhe Yan, Yiqiang Zhang, Bo Jiang, Chenghao Zhang, Borui Lu, Jiwen |
| contents | GUI grounding is a critical capability for vision-language models (VLMs) that enables automated interaction with graphical user interfaces by locating target elements from natural language instructions. However, grounding on GUI screenshots remains challenging due to high-resolution images, small UI elements, and ambiguous user instructions. In this work, we propose AdaZoom-GUI, an adaptive zoom-based GUI grounding framework that improves both localization accuracy and instruction understanding. Our approach introduces an instruction refinement module that rewrites natural language commands into explicit and detailed descriptions, allowing the grounding model to focus on precise element localization. In addition, we design a conditional zoom-in strategy that selectively performs a second-stage inference on predicted small elements, improving localization accuracy while avoiding unnecessary computation and context loss on simpler cases. To support this framework, we construct a high-quality GUI grounding dataset and train the grounding model using Group Relative Policy Optimization (GRPO), enabling the model to predict both click coordinates and element bounding boxes. Experiments on public benchmarks demonstrate that our method achieves state-of-the-art performance among models with comparable or even larger parameter sizes, highlighting its effectiveness for high-resolution GUI understanding and practical GUI agent deployment. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_17441 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | AdaZoom-GUI: Adaptive Zoom-based GUI Grounding with Instruction Refinement Pei, Siqi Tang, Liang Duan, Tiaonan Chen, Long Li, Shuxian Huang, Kaer Jing, Yanzhe Yan, Yiqiang Zhang, Bo Jiang, Chenghao Zhang, Borui Lu, Jiwen Computer Vision and Pattern Recognition Artificial Intelligence GUI grounding is a critical capability for vision-language models (VLMs) that enables automated interaction with graphical user interfaces by locating target elements from natural language instructions. However, grounding on GUI screenshots remains challenging due to high-resolution images, small UI elements, and ambiguous user instructions. In this work, we propose AdaZoom-GUI, an adaptive zoom-based GUI grounding framework that improves both localization accuracy and instruction understanding. Our approach introduces an instruction refinement module that rewrites natural language commands into explicit and detailed descriptions, allowing the grounding model to focus on precise element localization. In addition, we design a conditional zoom-in strategy that selectively performs a second-stage inference on predicted small elements, improving localization accuracy while avoiding unnecessary computation and context loss on simpler cases. To support this framework, we construct a high-quality GUI grounding dataset and train the grounding model using Group Relative Policy Optimization (GRPO), enabling the model to predict both click coordinates and element bounding boxes. Experiments on public benchmarks demonstrate that our method achieves state-of-the-art performance among models with comparable or even larger parameter sizes, highlighting its effectiveness for high-resolution GUI understanding and practical GUI agent deployment. |
| title | AdaZoom-GUI: Adaptive Zoom-based GUI Grounding with Instruction Refinement |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence |
| url | https://arxiv.org/abs/2603.17441 |