Scaling Computer-Use Grounding via User Interface Decomposition and Synthesis
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912669457448960 |
|---|---|
| author | Xie, Tianbao Deng, Jiaqi Li, Xiaochuan Yang, Junlin Wu, Haoyuan Chen, Jixuan Hu, Wenjing Wang, Xinyuan Xu, Yuhui Wang, Zekun Xu, Yiheng Wang, Junli Sahoo, Doyen Yu, Tao Xiong, Caiming |
| author_facet | Xie, Tianbao Deng, Jiaqi Li, Xiaochuan Yang, Junlin Wu, Haoyuan Chen, Jixuan Hu, Wenjing Wang, Xinyuan Xu, Yuhui Wang, Zekun Xu, Yiheng Wang, Junli Sahoo, Doyen Yu, Tao Xiong, Caiming |
| contents | Graphical user interface (GUI) grounding, the ability to map natural language instructions to specific actions on graphical user interfaces, remains a critical bottleneck in computer use agent development. Current benchmarks oversimplify grounding tasks as short referring expressions, failing to capture the complexity of real-world interactions that require software commonsense, layout understanding, and fine-grained manipulation capabilities. To address these limitations, we introduce OSWorld-G, a comprehensive benchmark comprising 564 finely annotated samples across diverse task types including text matching, element recognition, layout understanding, and precise manipulation. Additionally, we synthesize and release the largest computer use grounding dataset Jedi, which contains 4 million examples through multi-perspective decoupling of tasks. Our multi-scale models trained on Jedi demonstrate its effectiveness by outperforming existing approaches on ScreenSpot-v2, ScreenSpot-Pro, and our OSWorld-G. Furthermore, we demonstrate that improved grounding with Jedi directly enhances agentic capabilities of general foundation models on complex computer tasks, improving from 5% to 27% on OSWorld. Through detailed ablation studies, we identify key factors contributing to grounding performance and verify that combining specialized data for different interface elements enables compositional generalization to novel interfaces. All benchmark, data, checkpoints, and code are open-sourced and available at https://osworld-grounding.github.io. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_13227 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Scaling Computer-Use Grounding via User Interface Decomposition and Synthesis Xie, Tianbao Deng, Jiaqi Li, Xiaochuan Yang, Junlin Wu, Haoyuan Chen, Jixuan Hu, Wenjing Wang, Xinyuan Xu, Yuhui Wang, Zekun Xu, Yiheng Wang, Junli Sahoo, Doyen Yu, Tao Xiong, Caiming Artificial Intelligence Computation and Language Computer Vision and Pattern Recognition Human-Computer Interaction Graphical user interface (GUI) grounding, the ability to map natural language instructions to specific actions on graphical user interfaces, remains a critical bottleneck in computer use agent development. Current benchmarks oversimplify grounding tasks as short referring expressions, failing to capture the complexity of real-world interactions that require software commonsense, layout understanding, and fine-grained manipulation capabilities. To address these limitations, we introduce OSWorld-G, a comprehensive benchmark comprising 564 finely annotated samples across diverse task types including text matching, element recognition, layout understanding, and precise manipulation. Additionally, we synthesize and release the largest computer use grounding dataset Jedi, which contains 4 million examples through multi-perspective decoupling of tasks. Our multi-scale models trained on Jedi demonstrate its effectiveness by outperforming existing approaches on ScreenSpot-v2, ScreenSpot-Pro, and our OSWorld-G. Furthermore, we demonstrate that improved grounding with Jedi directly enhances agentic capabilities of general foundation models on complex computer tasks, improving from 5% to 27% on OSWorld. Through detailed ablation studies, we identify key factors contributing to grounding performance and verify that combining specialized data for different interface elements enables compositional generalization to novel interfaces. All benchmark, data, checkpoints, and code are open-sourced and available at https://osworld-grounding.github.io. |
| title | Scaling Computer-Use Grounding via User Interface Decomposition and Synthesis |
| topic | Artificial Intelligence Computation and Language Computer Vision and Pattern Recognition Human-Computer Interaction |
| url | https://arxiv.org/abs/2505.13227 |