Think Twice, Click Once: Enhancing GUI Grounding via Fast and Slow Systems
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912265830137856 |
|---|---|
| author | Tang, Fei Shen, Yongliang Zhang, Hang Chen, Siqi Hou, Guiyang Zhang, Wenqi Zhang, Wenqiao Song, Kaitao Lu, Weiming Zhuang, Yueting |
| author_facet | Tang, Fei Shen, Yongliang Zhang, Hang Chen, Siqi Hou, Guiyang Zhang, Wenqi Zhang, Wenqiao Song, Kaitao Lu, Weiming Zhuang, Yueting |
| contents | Humans can flexibly switch between different modes of thinking based on task complexity: from rapid intuitive judgments to in-depth analytical understanding. However, current Graphical User Interface (GUI) grounding systems which locate interface elements based on natural language instructions rely solely on immediate prediction without reasoning, struggling to understand complex interface layouts with nested structures and hierarchical relationships, limiting their effectiveness on complex interfaces. Inspired by human dual-system cognition, we present Focus, a novel GUI grounding framework that combines fast prediction with systematic analysis. The framework dynamically switches between rapid and deliberate processing through an adaptive system switching based on task complexity, optimizing both efficiency and accuracy. Focus decomposes grounding into progressive stages: interface summarization, visual focused analysis, and precise coordinate prediction. This structured decomposition enables systematic understanding of both interface layouts and visual relationships. Extensive experiments show that Focus achieves state-of-the-art performance using only 300K of the training data with a 2B parameter model compared to existing approaches. Focus demonstrates superior performance particularly in complex GUI scenarios, achieving 77.4% average accuracy on ScreenSpot and 13.3% on the more challenging ScreenSpot-Pro. Our analysis reveals the effectiveness of this dual-system approach while demonstrating its potential for improving complex GUI interaction scenarios. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_06470 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Think Twice, Click Once: Enhancing GUI Grounding via Fast and Slow Systems Tang, Fei Shen, Yongliang Zhang, Hang Chen, Siqi Hou, Guiyang Zhang, Wenqi Zhang, Wenqiao Song, Kaitao Lu, Weiming Zhuang, Yueting Artificial Intelligence Computation and Language Computer Vision and Pattern Recognition Humans can flexibly switch between different modes of thinking based on task complexity: from rapid intuitive judgments to in-depth analytical understanding. However, current Graphical User Interface (GUI) grounding systems which locate interface elements based on natural language instructions rely solely on immediate prediction without reasoning, struggling to understand complex interface layouts with nested structures and hierarchical relationships, limiting their effectiveness on complex interfaces. Inspired by human dual-system cognition, we present Focus, a novel GUI grounding framework that combines fast prediction with systematic analysis. The framework dynamically switches between rapid and deliberate processing through an adaptive system switching based on task complexity, optimizing both efficiency and accuracy. Focus decomposes grounding into progressive stages: interface summarization, visual focused analysis, and precise coordinate prediction. This structured decomposition enables systematic understanding of both interface layouts and visual relationships. Extensive experiments show that Focus achieves state-of-the-art performance using only 300K of the training data with a 2B parameter model compared to existing approaches. Focus demonstrates superior performance particularly in complex GUI scenarios, achieving 77.4% average accuracy on ScreenSpot and 13.3% on the more challenging ScreenSpot-Pro. Our analysis reveals the effectiveness of this dual-system approach while demonstrating its potential for improving complex GUI interaction scenarios. |
| title | Think Twice, Click Once: Enhancing GUI Grounding via Fast and Slow Systems |
| topic | Artificial Intelligence Computation and Language Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2503.06470 |