PG-Agent: An Agent Powered by Page Graph
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866912569217777664 |
|---|---|
| author | Chen, Weizhi Wang, Ziwei Yang, Leyang Zhou, Sheng Tang, Xiaoxuan Bu, Jiajun Li, Yong Jiang, Wei |
| author_facet | Chen, Weizhi Wang, Ziwei Yang, Leyang Zhou, Sheng Tang, Xiaoxuan Bu, Jiajun Li, Yong Jiang, Wei |
| contents | Graphical User Interface (GUI) agents possess significant commercial and social value, and GUI agents powered by advanced multimodal large language models (MLLMs) have demonstrated remarkable potential. Currently, existing GUI agents usually utilize sequential episodes of multi-step operations across pages as the prior GUI knowledge, which fails to capture the complex transition relationship between pages, making it challenging for the agents to deeply perceive the GUI environment and generalize to new scenarios. Therefore, we design an automated pipeline to transform the sequential episodes into page graphs, which explicitly model the graph structure of the pages that are naturally connected by actions. To fully utilize the page graphs, we further introduce Retrieval-Augmented Generation (RAG) technology to effectively retrieve reliable perception guidelines of GUI from them, and a tailored multi-agent framework PG-Agent with task decomposition strategy is proposed to be injected with the guidelines so that it can generalize to unseen scenarios. Extensive experiments on various benchmarks demonstrate the effectiveness of PG-Agent, even with limited episodes for page graph construction. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_03536 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | PG-Agent: An Agent Powered by Page Graph Chen, Weizhi Wang, Ziwei Yang, Leyang Zhou, Sheng Tang, Xiaoxuan Bu, Jiajun Li, Yong Jiang, Wei Artificial Intelligence Human-Computer Interaction Graphical User Interface (GUI) agents possess significant commercial and social value, and GUI agents powered by advanced multimodal large language models (MLLMs) have demonstrated remarkable potential. Currently, existing GUI agents usually utilize sequential episodes of multi-step operations across pages as the prior GUI knowledge, which fails to capture the complex transition relationship between pages, making it challenging for the agents to deeply perceive the GUI environment and generalize to new scenarios. Therefore, we design an automated pipeline to transform the sequential episodes into page graphs, which explicitly model the graph structure of the pages that are naturally connected by actions. To fully utilize the page graphs, we further introduce Retrieval-Augmented Generation (RAG) technology to effectively retrieve reliable perception guidelines of GUI from them, and a tailored multi-agent framework PG-Agent with task decomposition strategy is proposed to be injected with the guidelines so that it can generalize to unseen scenarios. Extensive experiments on various benchmarks demonstrate the effectiveness of PG-Agent, even with limited episodes for page graph construction. |
| title | PG-Agent: An Agent Powered by Page Graph |
| topic | Artificial Intelligence Human-Computer Interaction |
| url | https://arxiv.org/abs/2509.03536 |