MemGUI-Bench: Benchmarking Memory of Mobile GUI Agents in Dynamic Environments
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2026
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866917252097376256 |
|---|---|
| author | Liu, Guangyi Zhao, Pengxiang Liang, Yaozhen Luo, Qinyi Tang, Shunye Chai, Yuxiang Lin, Weifeng Xiao, Han Wang, WenHao Chen, Siheng Lu, Zhengxi Wu, Gao Wang, Hao Liu, Liang Liu, Yong |
| author_facet | Liu, Guangyi Zhao, Pengxiang Liang, Yaozhen Luo, Qinyi Tang, Shunye Chai, Yuxiang Lin, Weifeng Xiao, Han Wang, WenHao Chen, Siheng Lu, Zhengxi Wu, Gao Wang, Hao Liu, Liang Liu, Yong |
| contents | Current mobile GUI agent benchmarks systematically fail to assess memory capabilities, with only 5.2-11.8% memory-related tasks and no cross-session learning evaluation. We introduce MemGUI-Bench, a comprehensive memory-centric benchmark with pass@k and staged LLM-as-judge evaluation. Our contributions include: (1) a systematic memory taxonomy analyzing 11 agents across 5 architectures; (2) 128 tasks across 26 applications where 89.8% challenge memory through cross-temporal and cross-spatial retention; (3) MemGUI-Eval, an automated pipeline with Progressive Scrutiny and 7 hierarchical metrics; and (4) RQ-driven assessment of 11 state-of-the-art agents. Our experiments reveal significant memory deficits across all evaluated systems, identify 5 distinct failure modes, and synthesize 5 actionable design implications. All resources including code, benchmark, and evaluation results will be \textbf{\textit{fully open-sourced and continuously maintained}} at https://lgy0404.github.io/MemGUI-Bench/. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_06075 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | MemGUI-Bench: Benchmarking Memory of Mobile GUI Agents in Dynamic Environments Liu, Guangyi Zhao, Pengxiang Liang, Yaozhen Luo, Qinyi Tang, Shunye Chai, Yuxiang Lin, Weifeng Xiao, Han Wang, WenHao Chen, Siheng Lu, Zhengxi Wu, Gao Wang, Hao Liu, Liang Liu, Yong Distributed, Parallel, and Cluster Computing Current mobile GUI agent benchmarks systematically fail to assess memory capabilities, with only 5.2-11.8% memory-related tasks and no cross-session learning evaluation. We introduce MemGUI-Bench, a comprehensive memory-centric benchmark with pass@k and staged LLM-as-judge evaluation. Our contributions include: (1) a systematic memory taxonomy analyzing 11 agents across 5 architectures; (2) 128 tasks across 26 applications where 89.8% challenge memory through cross-temporal and cross-spatial retention; (3) MemGUI-Eval, an automated pipeline with Progressive Scrutiny and 7 hierarchical metrics; and (4) RQ-driven assessment of 11 state-of-the-art agents. Our experiments reveal significant memory deficits across all evaluated systems, identify 5 distinct failure modes, and synthesize 5 actionable design implications. All resources including code, benchmark, and evaluation results will be \textbf{\textit{fully open-sourced and continuously maintained}} at https://lgy0404.github.io/MemGUI-Bench/. |
| title | MemGUI-Bench: Benchmarking Memory of Mobile GUI Agents in Dynamic Environments |
| topic | Distributed, Parallel, and Cluster Computing |
| url | https://arxiv.org/abs/2602.06075 |