Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2506.14477 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908410742571008 |
|---|---|
| author | Yang, Jingqi Song, Zhilong Chen, Jiawei Song, Mingli Zhou, Sheng sun, linjun Ouyang, Xiaogang Chen, Chun Wang, Can |
| author_facet | Yang, Jingqi Song, Zhilong Chen, Jiawei Song, Mingli Zhou, Sheng sun, linjun Ouyang, Xiaogang Chen, Chun Wang, Can |
| contents | The development of high-quality datasets is crucial for benchmarking and advancing research in Graphical User Interface (GUI) agents. Despite their importance, existing datasets are often constructed under idealized conditions, overlooking the diverse anomalies frequently encountered in real-world deployments. To address this limitation, we introduce GUI-Robust, a novel dataset designed for comprehensive GUI agent evaluation, explicitly incorporating seven common types of anomalies observed in everyday GUI interactions. Furthermore, we propose a semi-automated dataset construction paradigm that collects user action sequences from natural interactions via RPA tools and then generate corresponding step and task descriptions for these actions with the assistance of MLLMs. This paradigm significantly reduces annotation time cost by a factor of over 19 times. Finally, we assess state-of-the-art GUI agents using the GUI-Robust dataset, revealing their substantial performance degradation in abnormal scenarios. We anticipate that our work will highlight the importance of robustness in GUI agents and inspires more future research in this direction. The dataset and code are available at https://github.com/chessbean1/GUI-Robust.. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_14477 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | GUI-Robust: A Comprehensive Dataset for Testing GUI Agent Robustness in Real-World Anomalies Yang, Jingqi Song, Zhilong Chen, Jiawei Song, Mingli Zhou, Sheng sun, linjun Ouyang, Xiaogang Chen, Chun Wang, Can Artificial Intelligence The development of high-quality datasets is crucial for benchmarking and advancing research in Graphical User Interface (GUI) agents. Despite their importance, existing datasets are often constructed under idealized conditions, overlooking the diverse anomalies frequently encountered in real-world deployments. To address this limitation, we introduce GUI-Robust, a novel dataset designed for comprehensive GUI agent evaluation, explicitly incorporating seven common types of anomalies observed in everyday GUI interactions. Furthermore, we propose a semi-automated dataset construction paradigm that collects user action sequences from natural interactions via RPA tools and then generate corresponding step and task descriptions for these actions with the assistance of MLLMs. This paradigm significantly reduces annotation time cost by a factor of over 19 times. Finally, we assess state-of-the-art GUI agents using the GUI-Robust dataset, revealing their substantial performance degradation in abnormal scenarios. We anticipate that our work will highlight the importance of robustness in GUI agents and inspires more future research in this direction. The dataset and code are available at https://github.com/chessbean1/GUI-Robust.. |
| title | GUI-Robust: A Comprehensive Dataset for Testing GUI Agent Robustness in Real-World Anomalies |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2506.14477 |