Saved in:
| Main Authors: | , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2508.18040 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916916212269056 |
|---|---|
| author | Wang, Xin Cui, Zhiyao Li, Hao Zeng, Ya Wang, Chenxu Song, Ruiqi Chen, Yihang Shao, Kun Zhang, Qiaosheng Liu, Jinzhuo Ren, Siyue Hu, Shuyue Wang, Zhen |
| author_facet | Wang, Xin Cui, Zhiyao Li, Hao Zeng, Ya Wang, Chenxu Song, Ruiqi Chen, Yihang Shao, Kun Zhang, Qiaosheng Liu, Jinzhuo Ren, Siyue Hu, Shuyue Wang, Zhen |
| contents | Vision language model (VLM)-based mobile agents show great potential for assisting users in performing instruction-driven tasks. However, these agents typically struggle with personalized instructions -- those containing ambiguous, user-specific context -- a challenge that has been largely overlooked in previous research. In this paper, we define personalized instructions and introduce PerInstruct, a novel human-annotated dataset covering diverse personalized instructions across various mobile scenarios. Furthermore, given the limited personalization capabilities of existing mobile agents, we propose PerPilot, a plug-and-play framework powered by large language models (LLMs) that enables mobile agents to autonomously perceive, understand, and execute personalized user instructions. PerPilot identifies personalized elements and autonomously completes instructions via two complementary approaches: memory-based retrieval and reasoning-based exploration. Experimental results demonstrate that PerPilot effectively handles personalized tasks with minimal user intervention and progressively improves its performance with continued use, underscoring the importance of personalization-aware reasoning for next-generation mobile agents. The dataset and code are available at: https://github.com/xinwang-nwpu/PerPilot |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_18040 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | PerPilot: Personalizing VLM-based Mobile Agents via Memory and Exploration Wang, Xin Cui, Zhiyao Li, Hao Zeng, Ya Wang, Chenxu Song, Ruiqi Chen, Yihang Shao, Kun Zhang, Qiaosheng Liu, Jinzhuo Ren, Siyue Hu, Shuyue Wang, Zhen Artificial Intelligence Vision language model (VLM)-based mobile agents show great potential for assisting users in performing instruction-driven tasks. However, these agents typically struggle with personalized instructions -- those containing ambiguous, user-specific context -- a challenge that has been largely overlooked in previous research. In this paper, we define personalized instructions and introduce PerInstruct, a novel human-annotated dataset covering diverse personalized instructions across various mobile scenarios. Furthermore, given the limited personalization capabilities of existing mobile agents, we propose PerPilot, a plug-and-play framework powered by large language models (LLMs) that enables mobile agents to autonomously perceive, understand, and execute personalized user instructions. PerPilot identifies personalized elements and autonomously completes instructions via two complementary approaches: memory-based retrieval and reasoning-based exploration. Experimental results demonstrate that PerPilot effectively handles personalized tasks with minimal user intervention and progressively improves its performance with continued use, underscoring the importance of personalization-aware reasoning for next-generation mobile agents. The dataset and code are available at: https://github.com/xinwang-nwpu/PerPilot |
| title | PerPilot: Personalizing VLM-based Mobile Agents via Memory and Exploration |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2508.18040 |