Mobile-Env: Building Qualified Evaluation Benchmarks for LLM-GUI Interaction
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917691892170752 |
|---|---|
| author | Zhang, Danyang Shen, Zhennan Xie, Rui Zhang, Situo Xie, Tianbao Zhao, Zihan Chen, Siyuan Chen, Lu Xu, Hongshen Cao, Ruisheng Yu, Kai |
| author_facet | Zhang, Danyang Shen, Zhennan Xie, Rui Zhang, Situo Xie, Tianbao Zhao, Zihan Chen, Siyuan Chen, Lu Xu, Hongshen Cao, Ruisheng Yu, Kai |
| contents | The Graphical User Interface (GUI) is pivotal for human interaction with the digital world, enabling efficient device control and the completion of complex tasks. Recent progress in Large Language Models (LLMs) and Vision Language Models (VLMs) offers the chance to create advanced GUI agents. To ensure their effectiveness, there's a pressing need for qualified benchmarks that provide trustworthy and reproducible evaluations -- a challenge current benchmarks often fail to address. To tackle this issue, we introduce Mobile-Env, a comprehensive toolkit tailored for creating GUI benchmarks in the Android mobile environment. Mobile-Env offers an isolated and controllable setting for reliable evaluations, and accommodates intermediate instructions and rewards to reflect real-world usage more naturally. Utilizing Mobile-Env, we collect an open-world task set across various real-world apps and a fixed world set, WikiHow, which captures a significant amount of dynamic online contents for fully controllable and reproducible evaluation. We conduct comprehensive evaluations of LLM agents using these benchmarks. Our findings reveal that even advanced models (e.g., GPT-4V and LLaMA-3) struggle with tasks that are relatively simple for humans. This highlights a crucial gap in current models and underscores the importance of developing more capable foundation models and more effective GUI agent frameworks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2305_08144 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Mobile-Env: Building Qualified Evaluation Benchmarks for LLM-GUI Interaction Zhang, Danyang Shen, Zhennan Xie, Rui Zhang, Situo Xie, Tianbao Zhao, Zihan Chen, Siyuan Chen, Lu Xu, Hongshen Cao, Ruisheng Yu, Kai Artificial Intelligence The Graphical User Interface (GUI) is pivotal for human interaction with the digital world, enabling efficient device control and the completion of complex tasks. Recent progress in Large Language Models (LLMs) and Vision Language Models (VLMs) offers the chance to create advanced GUI agents. To ensure their effectiveness, there's a pressing need for qualified benchmarks that provide trustworthy and reproducible evaluations -- a challenge current benchmarks often fail to address. To tackle this issue, we introduce Mobile-Env, a comprehensive toolkit tailored for creating GUI benchmarks in the Android mobile environment. Mobile-Env offers an isolated and controllable setting for reliable evaluations, and accommodates intermediate instructions and rewards to reflect real-world usage more naturally. Utilizing Mobile-Env, we collect an open-world task set across various real-world apps and a fixed world set, WikiHow, which captures a significant amount of dynamic online contents for fully controllable and reproducible evaluation. We conduct comprehensive evaluations of LLM agents using these benchmarks. Our findings reveal that even advanced models (e.g., GPT-4V and LLaMA-3) struggle with tasks that are relatively simple for humans. This highlights a crucial gap in current models and underscores the importance of developing more capable foundation models and more effective GUI agent frameworks. |
| title | Mobile-Env: Building Qualified Evaluation Benchmarks for LLM-GUI Interaction |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2305.08144 |