Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2504.07981 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915236750032896 |
|---|---|
| author | Li, Kaixin Meng, Ziyang Lin, Hongzhan Luo, Ziyang Tian, Yuchen Ma, Jing Huang, Zhiyong Chua, Tat-Seng |
| author_facet | Li, Kaixin Meng, Ziyang Lin, Hongzhan Luo, Ziyang Tian, Yuchen Ma, Jing Huang, Zhiyong Chua, Tat-Seng |
| contents | Recent advancements in Multi-modal Large Language Models (MLLMs) have led to significant progress in developing GUI agents for general tasks such as web browsing and mobile phone use. However, their application in professional domains remains under-explored. These specialized workflows introduce unique challenges for GUI perception models, including high-resolution displays, smaller target sizes, and complex environments. In this paper, we introduce ScreenSpot-Pro, a new benchmark designed to rigorously evaluate the grounding capabilities of MLLMs in high-resolution professional settings. The benchmark comprises authentic high-resolution images from a variety of professional domains with expert annotations. It spans 23 applications across five industries and three operating systems. Existing GUI grounding models perform poorly on this dataset, with the best model achieving only 18.9%. Our experiments reveal that strategically reducing the search area enhances accuracy. Based on this insight, we propose ScreenSeekeR, a visual search method that utilizes the GUI knowledge of a strong planner to guide a cascaded search, achieving state-of-the-art performance with 48.1% without any additional training. We hope that our benchmark and findings will advance the development of GUI agents for professional applications. Code, data and leaderboard can be found at https://gui-agent.github.io/grounding-leaderboard. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_07981 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Use Li, Kaixin Meng, Ziyang Lin, Hongzhan Luo, Ziyang Tian, Yuchen Ma, Jing Huang, Zhiyong Chua, Tat-Seng Computer Vision and Pattern Recognition Human-Computer Interaction Multimedia 68-11 68-04 I.2.7; I.2.10 Recent advancements in Multi-modal Large Language Models (MLLMs) have led to significant progress in developing GUI agents for general tasks such as web browsing and mobile phone use. However, their application in professional domains remains under-explored. These specialized workflows introduce unique challenges for GUI perception models, including high-resolution displays, smaller target sizes, and complex environments. In this paper, we introduce ScreenSpot-Pro, a new benchmark designed to rigorously evaluate the grounding capabilities of MLLMs in high-resolution professional settings. The benchmark comprises authentic high-resolution images from a variety of professional domains with expert annotations. It spans 23 applications across five industries and three operating systems. Existing GUI grounding models perform poorly on this dataset, with the best model achieving only 18.9%. Our experiments reveal that strategically reducing the search area enhances accuracy. Based on this insight, we propose ScreenSeekeR, a visual search method that utilizes the GUI knowledge of a strong planner to guide a cascaded search, achieving state-of-the-art performance with 48.1% without any additional training. We hope that our benchmark and findings will advance the development of GUI agents for professional applications. Code, data and leaderboard can be found at https://gui-agent.github.io/grounding-leaderboard. |
| title | ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Use |
| topic | Computer Vision and Pattern Recognition Human-Computer Interaction Multimedia 68-11 68-04 I.2.7; I.2.10 |
| url | https://arxiv.org/abs/2504.07981 |