MP-GUI: Modality Perception with MLLMs for GUI Understanding
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909541025710080 |
|---|---|
| author | Wang, Ziwei Chen, Weizhi Yang, Leyang Zhou, Sheng Zhao, Shengchu Zhan, Hanbei Jin, Jiongchao Li, Liangcheng Shao, Zirui Bu, Jiajun |
| author_facet | Wang, Ziwei Chen, Weizhi Yang, Leyang Zhou, Sheng Zhao, Shengchu Zhan, Hanbei Jin, Jiongchao Li, Liangcheng Shao, Zirui Bu, Jiajun |
| contents | Graphical user interface (GUI) has become integral to modern society, making it crucial to be understood for human-centric systems. However, unlike natural images or documents, GUIs comprise artificially designed graphical elements arranged to convey specific semantic meanings. Current multi-modal large language models (MLLMs) already proficient in processing graphical and textual components suffer from hurdles in GUI understanding due to the lack of explicit spatial structure modeling. Moreover, obtaining high-quality spatial structure data is challenging due to privacy issues and noisy environments. To address these challenges, we present MP-GUI, a specially designed MLLM for GUI understanding. MP-GUI features three precisely specialized perceivers to extract graphical, textual, and spatial modalities from the screen as GUI-tailored visual clues, with spatial structure refinement strategy and adaptively combined via a fusion gate to meet the specific preferences of different GUI understanding tasks. To cope with the scarcity of training data, we also introduce a pipeline for automatically data collecting. Extensive experiments demonstrate that MP-GUI achieves impressive results on various GUI understanding tasks with limited data. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_14021 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | MP-GUI: Modality Perception with MLLMs for GUI Understanding Wang, Ziwei Chen, Weizhi Yang, Leyang Zhou, Sheng Zhao, Shengchu Zhan, Hanbei Jin, Jiongchao Li, Liangcheng Shao, Zirui Bu, Jiajun Computer Vision and Pattern Recognition Artificial Intelligence Human-Computer Interaction Graphical user interface (GUI) has become integral to modern society, making it crucial to be understood for human-centric systems. However, unlike natural images or documents, GUIs comprise artificially designed graphical elements arranged to convey specific semantic meanings. Current multi-modal large language models (MLLMs) already proficient in processing graphical and textual components suffer from hurdles in GUI understanding due to the lack of explicit spatial structure modeling. Moreover, obtaining high-quality spatial structure data is challenging due to privacy issues and noisy environments. To address these challenges, we present MP-GUI, a specially designed MLLM for GUI understanding. MP-GUI features three precisely specialized perceivers to extract graphical, textual, and spatial modalities from the screen as GUI-tailored visual clues, with spatial structure refinement strategy and adaptively combined via a fusion gate to meet the specific preferences of different GUI understanding tasks. To cope with the scarcity of training data, we also introduce a pipeline for automatically data collecting. Extensive experiments demonstrate that MP-GUI achieves impressive results on various GUI understanding tasks with limited data. |
| title | MP-GUI: Modality Perception with MLLMs for GUI Understanding |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Human-Computer Interaction |
| url | https://arxiv.org/abs/2503.14021 |