KG-RAG: Enhancing GUI Agent Decision-Making via Knowledge Graph-Driven Retrieval-Augmented Generation
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912560160178176 |
|---|---|
| author | Guan, Ziyi Li, Jason Chun Lok Hou, Zhijian Zhang, Pingping Xu, Donglai Zhao, Yuzhi Wu, Mengyang Chen, Jinpeng Nguyen, Thanh-Toan Xian, Pengfei Ma, Wenao Qin, Shengchao Chesi, Graziano Wong, Ngai |
| author_facet | Guan, Ziyi Li, Jason Chun Lok Hou, Zhijian Zhang, Pingping Xu, Donglai Zhao, Yuzhi Wu, Mengyang Chen, Jinpeng Nguyen, Thanh-Toan Xian, Pengfei Ma, Wenao Qin, Shengchao Chesi, Graziano Wong, Ngai |
| contents | Despite recent progress, Graphic User Interface (GUI) agents powered by Large Language Models (LLMs) struggle with complex mobile tasks due to limited app-specific knowledge. While UI Transition Graphs (UTGs) offer structured navigation representations, they are underutilized due to poor extraction and inefficient integration. We introduce KG-RAG, a Knowledge Graph-driven Retrieval-Augmented Generation framework that transforms fragmented UTGs into structured vector databases for efficient real-time retrieval. By leveraging an intent-guided LLM search method, KG-RAG generates actionable navigation paths, enhancing agent decision-making. Experiments across diverse mobile apps show that KG-RAG outperforms existing methods, achieving a 75.8% success rate (8.9% improvement over AutoDroid), 84.6% decision accuracy (8.1% improvement), and reducing average task steps from 4.5 to 4.1. Additionally, we present KG-Android-Bench and KG-Harmony-Bench, two benchmarks tailored to the Chinese mobile ecosystem for future research. Finally, KG-RAG transfers to web/desktop (+40% SR on Weibo-web; +20% on QQ Music-desktop), and a UTG cost ablation shows accuracy saturates at ~4h per complex app, enabling practical deployment trade-offs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_00366 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | KG-RAG: Enhancing GUI Agent Decision-Making via Knowledge Graph-Driven Retrieval-Augmented Generation Guan, Ziyi Li, Jason Chun Lok Hou, Zhijian Zhang, Pingping Xu, Donglai Zhao, Yuzhi Wu, Mengyang Chen, Jinpeng Nguyen, Thanh-Toan Xian, Pengfei Ma, Wenao Qin, Shengchao Chesi, Graziano Wong, Ngai Multiagent Systems Computation and Language Multimedia Despite recent progress, Graphic User Interface (GUI) agents powered by Large Language Models (LLMs) struggle with complex mobile tasks due to limited app-specific knowledge. While UI Transition Graphs (UTGs) offer structured navigation representations, they are underutilized due to poor extraction and inefficient integration. We introduce KG-RAG, a Knowledge Graph-driven Retrieval-Augmented Generation framework that transforms fragmented UTGs into structured vector databases for efficient real-time retrieval. By leveraging an intent-guided LLM search method, KG-RAG generates actionable navigation paths, enhancing agent decision-making. Experiments across diverse mobile apps show that KG-RAG outperforms existing methods, achieving a 75.8% success rate (8.9% improvement over AutoDroid), 84.6% decision accuracy (8.1% improvement), and reducing average task steps from 4.5 to 4.1. Additionally, we present KG-Android-Bench and KG-Harmony-Bench, two benchmarks tailored to the Chinese mobile ecosystem for future research. Finally, KG-RAG transfers to web/desktop (+40% SR on Weibo-web; +20% on QQ Music-desktop), and a UTG cost ablation shows accuracy saturates at ~4h per complex app, enabling practical deployment trade-offs. |
| title | KG-RAG: Enhancing GUI Agent Decision-Making via Knowledge Graph-Driven Retrieval-Augmented Generation |
| topic | Multiagent Systems Computation and Language Multimedia |
| url | https://arxiv.org/abs/2509.00366 |