AmbiBench: Benchmarking Mobile GUI Agents Beyond One-Shot Instructions in the Wild
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2026
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866910020126375936 |
|---|---|
| author | Sun, Jiazheng Li, Mingxuan Zhang, Yingying Niu, Jiayang Wu, Yachen Jin, Ruihan Lei, Shuyu Tan, Pengrongrui Zhang, Zongyu Wang, Ruoyi Yang, Jiachen Yang, Boyu Liu, Jiacheng Peng, Xin |
| author_facet | Sun, Jiazheng Li, Mingxuan Zhang, Yingying Niu, Jiayang Wu, Yachen Jin, Ruihan Lei, Shuyu Tan, Pengrongrui Zhang, Zongyu Wang, Ruoyi Yang, Jiachen Yang, Boyu Liu, Jiacheng Peng, Xin |
| contents | Benchmarks are paramount for gauging progress in the domain of Mobile GUI Agents. In practical scenarios, users frequently fail to articulate precise directives containing full task details at the onset, and their expressions are typically ambiguous. Consequently, agents are required to converge on the user's true intent via active clarification and interaction during execution. However, existing benchmarks predominantly operate under the idealized assumption that user-issued instructions are complete and unequivocal. This paradigm focuses exclusively on assessing single-turn execution while overlooking the alignment capability of the agent. To address this limitation, we introduce AmbiBench, the first benchmark incorporating a taxonomy of instruction clarity to shift evaluation from unidirectional instruction following to bidirectional intent alignment. Grounded in Cognitive Gap theory, we propose a taxonomy of four clarity levels: Detailed, Standard, Incomplete, and Ambiguous. We construct a rigorous dataset of 240 ecologically valid tasks across 25 applications, subject to strict review protocols. Furthermore, targeting evaluation in dynamic environments, we develop MUSE (Mobile User Satisfaction Evaluator), an automated framework utilizing an MLLM-as-a-judge multi-agent architecture. MUSE performs fine-grained auditing across three dimensions: Outcome Effectiveness, Execution Quality, and Interaction Quality. Empirical results on AmbiBench reveal the performance boundaries of SoTA agents across different clarity levels, quantify the gains derived from active interaction, and validate the strong correlation between MUSE and human judgment. This work redefines evaluation standards, laying the foundation for next-generation agents capable of truly understanding user intent. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_11750 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | AmbiBench: Benchmarking Mobile GUI Agents Beyond One-Shot Instructions in the Wild Sun, Jiazheng Li, Mingxuan Zhang, Yingying Niu, Jiayang Wu, Yachen Jin, Ruihan Lei, Shuyu Tan, Pengrongrui Zhang, Zongyu Wang, Ruoyi Yang, Jiachen Yang, Boyu Liu, Jiacheng Peng, Xin Software Engineering Artificial Intelligence Human-Computer Interaction Benchmarks are paramount for gauging progress in the domain of Mobile GUI Agents. In practical scenarios, users frequently fail to articulate precise directives containing full task details at the onset, and their expressions are typically ambiguous. Consequently, agents are required to converge on the user's true intent via active clarification and interaction during execution. However, existing benchmarks predominantly operate under the idealized assumption that user-issued instructions are complete and unequivocal. This paradigm focuses exclusively on assessing single-turn execution while overlooking the alignment capability of the agent. To address this limitation, we introduce AmbiBench, the first benchmark incorporating a taxonomy of instruction clarity to shift evaluation from unidirectional instruction following to bidirectional intent alignment. Grounded in Cognitive Gap theory, we propose a taxonomy of four clarity levels: Detailed, Standard, Incomplete, and Ambiguous. We construct a rigorous dataset of 240 ecologically valid tasks across 25 applications, subject to strict review protocols. Furthermore, targeting evaluation in dynamic environments, we develop MUSE (Mobile User Satisfaction Evaluator), an automated framework utilizing an MLLM-as-a-judge multi-agent architecture. MUSE performs fine-grained auditing across three dimensions: Outcome Effectiveness, Execution Quality, and Interaction Quality. Empirical results on AmbiBench reveal the performance boundaries of SoTA agents across different clarity levels, quantify the gains derived from active interaction, and validate the strong correlation between MUSE and human judgment. This work redefines evaluation standards, laying the foundation for next-generation agents capable of truly understanding user intent. |
| title | AmbiBench: Benchmarking Mobile GUI Agents Beyond One-Shot Instructions in the Wild |
| topic | Software Engineering Artificial Intelligence Human-Computer Interaction |
| url | https://arxiv.org/abs/2602.11750 |