Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2511.04307 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912697447088128 |
|---|---|
| author | Mu, Jian Zhang, Chaoyun Ni, Chiming Wang, Lu Qiao, Bo Mathur, Kartik Wu, Qianhui Xie, Yuhang Ma, Xiaojun Zhou, Mengyu Qin, Si Li, Liqun Kang, Yu Ma, Minghua Lin, Qingwei Rajmohan, Saravan Zhang, Dongmei |
| author_facet | Mu, Jian Zhang, Chaoyun Ni, Chiming Wang, Lu Qiao, Bo Mathur, Kartik Wu, Qianhui Xie, Yuhang Ma, Xiaojun Zhou, Mengyu Qin, Si Li, Liqun Kang, Yu Ma, Minghua Lin, Qingwei Rajmohan, Saravan Zhang, Dongmei |
| contents | We introduce GUI-360$^\circ$, a large-scale, comprehensive dataset and benchmark suite designed to advance computer-using agents (CUAs). CUAs present unique challenges and is constrained by three persistent gaps: a scarcity of real-world CUA tasks, the lack of automated collection-and-annotation pipelines for multi-modal trajectories, and the absence of a unified benchmark that jointly evaluates GUI grounding, screen parsing, and action prediction.
GUI-360$^\circ$ addresses these gaps with an LLM-augmented, largely automated pipeline for query sourcing, environment-template construction, task instantiation, batched execution, and LLM-driven quality filtering. The released corpus contains over 1.2M executed action steps across thousands of trajectories in popular Windows office applications, and includes full-resolution screenshots, accessibility metadata when available, instantiated goals, intermediate reasoning traces, and both successful and failed action trajectories. The dataset supports three canonical tasks, GUI grounding, screen parsing, and action prediction, and a hybrid GUI+API action space that reflects modern agent designs. Benchmarking state-of-the-art vision--language models on GUI-360$^\circ$ reveals substantial out-of-the-box shortcomings in grounding and action prediction; supervised fine-tuning and reinforcement learning yield significant gains but do not close the gap to human-level reliability. We release GUI-360$^\circ$ and accompanying code to facilitate reproducible research and accelerate progress on robust desktop CUAs.
The full dataset has been made public on https://huggingface.co/datasets/vyokky/GUI-360. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_04307 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | GUI-360$^\circ$: A Comprehensive Dataset and Benchmark for Computer-Using Agents Mu, Jian Zhang, Chaoyun Ni, Chiming Wang, Lu Qiao, Bo Mathur, Kartik Wu, Qianhui Xie, Yuhang Ma, Xiaojun Zhou, Mengyu Qin, Si Li, Liqun Kang, Yu Ma, Minghua Lin, Qingwei Rajmohan, Saravan Zhang, Dongmei Artificial Intelligence We introduce GUI-360$^\circ$, a large-scale, comprehensive dataset and benchmark suite designed to advance computer-using agents (CUAs). CUAs present unique challenges and is constrained by three persistent gaps: a scarcity of real-world CUA tasks, the lack of automated collection-and-annotation pipelines for multi-modal trajectories, and the absence of a unified benchmark that jointly evaluates GUI grounding, screen parsing, and action prediction. GUI-360$^\circ$ addresses these gaps with an LLM-augmented, largely automated pipeline for query sourcing, environment-template construction, task instantiation, batched execution, and LLM-driven quality filtering. The released corpus contains over 1.2M executed action steps across thousands of trajectories in popular Windows office applications, and includes full-resolution screenshots, accessibility metadata when available, instantiated goals, intermediate reasoning traces, and both successful and failed action trajectories. The dataset supports three canonical tasks, GUI grounding, screen parsing, and action prediction, and a hybrid GUI+API action space that reflects modern agent designs. Benchmarking state-of-the-art vision--language models on GUI-360$^\circ$ reveals substantial out-of-the-box shortcomings in grounding and action prediction; supervised fine-tuning and reinforcement learning yield significant gains but do not close the gap to human-level reliability. We release GUI-360$^\circ$ and accompanying code to facilitate reproducible research and accelerate progress on robust desktop CUAs. The full dataset has been made public on https://huggingface.co/datasets/vyokky/GUI-360. |
| title | GUI-360$^\circ$: A Comprehensive Dataset and Benchmark for Computer-Using Agents |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2511.04307 |