Beyond Pass or Fail: Multi-Dimensional Benchmarking of Foundation Models for Goal-based Mobile UI Navigation
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866910821550915584 |
|---|---|
| author | Ran, Dezhi Wu, Mengzhou Yu, Hao Li, Yuetong Ren, Jun Cao, Yuan Zeng, Xia Lu, Haochuan Xu, Zexin Xu, Mengqian Su, Ting Yao, Liangchao Xiong, Ting Yang, Wei Deng, Yuetang Marron, Assaf Harel, David Xie, Tao |
| author_facet | Ran, Dezhi Wu, Mengzhou Yu, Hao Li, Yuetong Ren, Jun Cao, Yuan Zeng, Xia Lu, Haochuan Xu, Zexin Xu, Mengqian Su, Ting Yao, Liangchao Xiong, Ting Yang, Wei Deng, Yuetang Marron, Assaf Harel, David Xie, Tao |
| contents | Recent advances of foundation models (FMs) have made navigating mobile applications (apps) based on high-level goal instructions within reach, with significant industrial applications such as UI testing. While existing benchmarks evaluate FM-based UI navigation using the binary pass/fail metric, they have two major limitations: they cannot reflect the complex nature of mobile UI navigation where FMs may fail for various reasons (e.g., misunderstanding instructions and failed planning), and they lack industrial relevance due to oversimplified tasks that poorly represent real-world scenarios. To address the preceding limitations, we propose Sphinx, a comprehensive benchmark for multi-dimensional evaluation of FMs in industrial settings of UI navigation. Sphinx introduces a specialized toolkit that evaluates five essential FM capabilities, providing detailed insights into failure modes such as insufficient app knowledge or planning issues. Using both popular Google Play applications and WeChat's internal UI test cases, we evaluate 8 FMs with 20 different configurations. Our results show that existing FMs universally struggle with goal-based testing tasks, primarily due to insufficient UI-specific capabilities. We summarize seven lessons learned from benchmarking FMs with Sphinx, providing clear directions for improving FM-based mobile UI navigation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_02863 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Beyond Pass or Fail: Multi-Dimensional Benchmarking of Foundation Models for Goal-based Mobile UI Navigation Ran, Dezhi Wu, Mengzhou Yu, Hao Li, Yuetong Ren, Jun Cao, Yuan Zeng, Xia Lu, Haochuan Xu, Zexin Xu, Mengqian Su, Ting Yao, Liangchao Xiong, Ting Yang, Wei Deng, Yuetang Marron, Assaf Harel, David Xie, Tao Software Engineering Recent advances of foundation models (FMs) have made navigating mobile applications (apps) based on high-level goal instructions within reach, with significant industrial applications such as UI testing. While existing benchmarks evaluate FM-based UI navigation using the binary pass/fail metric, they have two major limitations: they cannot reflect the complex nature of mobile UI navigation where FMs may fail for various reasons (e.g., misunderstanding instructions and failed planning), and they lack industrial relevance due to oversimplified tasks that poorly represent real-world scenarios. To address the preceding limitations, we propose Sphinx, a comprehensive benchmark for multi-dimensional evaluation of FMs in industrial settings of UI navigation. Sphinx introduces a specialized toolkit that evaluates five essential FM capabilities, providing detailed insights into failure modes such as insufficient app knowledge or planning issues. Using both popular Google Play applications and WeChat's internal UI test cases, we evaluate 8 FMs with 20 different configurations. Our results show that existing FMs universally struggle with goal-based testing tasks, primarily due to insufficient UI-specific capabilities. We summarize seven lessons learned from benchmarking FMs with Sphinx, providing clear directions for improving FM-based mobile UI navigation. |
| title | Beyond Pass or Fail: Multi-Dimensional Benchmarking of Foundation Models for Goal-based Mobile UI Navigation |
| topic | Software Engineering |
| url | https://arxiv.org/abs/2501.02863 |