MobA: Multifaceted Memory-Enhanced Adaptive Planning for Efficient Mobile Task Automation
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866915604907163648 |
|---|---|
| author | Zhu, Zichen Tang, Hao Li, Yansi Liu, Dingye Xu, Hongshen Lan, Kunyao Zhang, Danyang Jiang, Yixuan Zhou, Hao Wang, Chenrun Zhang, Situo Sun, Liangtai Wang, Yixiao Sun, Yuheng Chen, Lu Yu, Kai |
| author_facet | Zhu, Zichen Tang, Hao Li, Yansi Liu, Dingye Xu, Hongshen Lan, Kunyao Zhang, Danyang Jiang, Yixuan Zhou, Hao Wang, Chenrun Zhang, Situo Sun, Liangtai Wang, Yixiao Sun, Yuheng Chen, Lu Yu, Kai |
| contents | Existing Multimodal Large Language Model (MLLM)-based agents face significant challenges in handling complex GUI (Graphical User Interface) interactions on devices. These challenges arise from the dynamic and structured nature of GUI environments, which integrate text, images, and spatial relationships, as well as the variability in action spaces across different pages and tasks. To address these limitations, we propose MobA, a novel MLLM-based mobile assistant system. MobA introduces an adaptive planning module that incorporates a reflection mechanism for error recovery and dynamically adjusts plans to align with the real environment contexts and action module's execution capacity. Additionally, a multifaceted memory module provides comprehensive memory support to enhance adaptability and efficiency. We also present MobBench, a dataset designed for complex mobile interactions. Experimental results on MobBench and AndroidArena demonstrate MobA's ability to handle dynamic GUI environments and perform complex mobile tasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_13757 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | MobA: Multifaceted Memory-Enhanced Adaptive Planning for Efficient Mobile Task Automation Zhu, Zichen Tang, Hao Li, Yansi Liu, Dingye Xu, Hongshen Lan, Kunyao Zhang, Danyang Jiang, Yixuan Zhou, Hao Wang, Chenrun Zhang, Situo Sun, Liangtai Wang, Yixiao Sun, Yuheng Chen, Lu Yu, Kai Multiagent Systems Artificial Intelligence Computation and Language Human-Computer Interaction Existing Multimodal Large Language Model (MLLM)-based agents face significant challenges in handling complex GUI (Graphical User Interface) interactions on devices. These challenges arise from the dynamic and structured nature of GUI environments, which integrate text, images, and spatial relationships, as well as the variability in action spaces across different pages and tasks. To address these limitations, we propose MobA, a novel MLLM-based mobile assistant system. MobA introduces an adaptive planning module that incorporates a reflection mechanism for error recovery and dynamically adjusts plans to align with the real environment contexts and action module's execution capacity. Additionally, a multifaceted memory module provides comprehensive memory support to enhance adaptability and efficiency. We also present MobBench, a dataset designed for complex mobile interactions. Experimental results on MobBench and AndroidArena demonstrate MobA's ability to handle dynamic GUI environments and perform complex mobile tasks. |
| title | MobA: Multifaceted Memory-Enhanced Adaptive Planning for Efficient Mobile Task Automation |
| topic | Multiagent Systems Artificial Intelligence Computation and Language Human-Computer Interaction |
| url | https://arxiv.org/abs/2410.13757 |