GUI Knowledge Bench: Revealing the Knowledge Gap of VLMs in GUI Tasks
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908821730885632 |
|---|---|
| author | Shi, Chenrui Yu, Zedong Gao, Zhi Feng, Ruining Liu, Enqi Wu, Yuwei Jia, Yunde Xiang, Liuyu He, Zhaofeng Li, Qing |
| author_facet | Shi, Chenrui Yu, Zedong Gao, Zhi Feng, Ruining Liu, Enqi Wu, Yuwei Jia, Yunde Xiang, Liuyu He, Zhaofeng Li, Qing |
| contents | Vision language models (VLMs) have advanced graphical user interface (GUI) task automation but still lag behind humans. We hypothesize this gap stems from missing core GUI knowledge, which existing training schemes (such as supervised fine tuning and reinforcement learning) alone cannot fully address. By analyzing common failure patterns in GUI task execution, we distill GUI knowledge into three dimensions: (1) interface knowledge about widget functions, layout semantics, and system states; (2) interaction knowledge about GUI interaction types and effects; and (3) procedure knowledge of task objectives and workflow sequences. We further introduce GUI Knowledge Bench, a benchmark with multiple-choice and yes/no questions across six platforms (Web, Android, MacOS, Windows, Linux, IOS) and 292 applications. Our evaluation indicates that current VLMs are generally aware of the functions of individual widgets, but lack the GUI-specific knowledge required to track system states, adhere to GUI interaction conventions, and assess task completion progress. Experiments on real-world GUI tasks further validate the close link between GUI knowledge and task success. By providing a structured framework for assessing GUI knowledge, our work supports the selection of VLMs with greater potential prior to downstream training and provides insights for building more capable GUI agents. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_26098 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | GUI Knowledge Bench: Revealing the Knowledge Gap of VLMs in GUI Tasks Shi, Chenrui Yu, Zedong Gao, Zhi Feng, Ruining Liu, Enqi Wu, Yuwei Jia, Yunde Xiang, Liuyu He, Zhaofeng Li, Qing Artificial Intelligence Vision language models (VLMs) have advanced graphical user interface (GUI) task automation but still lag behind humans. We hypothesize this gap stems from missing core GUI knowledge, which existing training schemes (such as supervised fine tuning and reinforcement learning) alone cannot fully address. By analyzing common failure patterns in GUI task execution, we distill GUI knowledge into three dimensions: (1) interface knowledge about widget functions, layout semantics, and system states; (2) interaction knowledge about GUI interaction types and effects; and (3) procedure knowledge of task objectives and workflow sequences. We further introduce GUI Knowledge Bench, a benchmark with multiple-choice and yes/no questions across six platforms (Web, Android, MacOS, Windows, Linux, IOS) and 292 applications. Our evaluation indicates that current VLMs are generally aware of the functions of individual widgets, but lack the GUI-specific knowledge required to track system states, adhere to GUI interaction conventions, and assess task completion progress. Experiments on real-world GUI tasks further validate the close link between GUI knowledge and task success. By providing a structured framework for assessing GUI knowledge, our work supports the selection of VLMs with greater potential prior to downstream training and provides insights for building more capable GUI agents. |
| title | GUI Knowledge Bench: Revealing the Knowledge Gap of VLMs in GUI Tasks |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2510.26098 |