Ferret-UI 2: Mastering Universal User Interface Understanding Across Platforms
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915176203157504 |
|---|---|
| author | Li, Zhangheng You, Keen Zhang, Haotian Feng, Di Agrawal, Harsh Li, Xiujun Moorthy, Mohana Prasad Sathya Nichols, Jeff Yang, Yinfei Gan, Zhe |
| author_facet | Li, Zhangheng You, Keen Zhang, Haotian Feng, Di Agrawal, Harsh Li, Xiujun Moorthy, Mohana Prasad Sathya Nichols, Jeff Yang, Yinfei Gan, Zhe |
| contents | Building a generalist model for user interface (UI) understanding is challenging due to various foundational issues, such as platform diversity, resolution variation, and data limitation. In this paper, we introduce Ferret-UI 2, a multimodal large language model (MLLM) designed for universal UI understanding across a wide range of platforms, including iPhone, Android, iPad, Webpage, and AppleTV. Building on the foundation of Ferret-UI, Ferret-UI 2 introduces three key innovations: support for multiple platform types, high-resolution perception through adaptive scaling, and advanced task training data generation powered by GPT-4o with set-of-mark visual prompting. These advancements enable Ferret-UI 2 to perform complex, user-centered interactions, making it highly versatile and adaptable for the expanding diversity of platform ecosystems. Extensive empirical experiments on referring, grounding, user-centric advanced tasks (comprising 9 subtasks $\times$ 5 platforms), GUIDE next-action prediction dataset, and GUI-World multi-platform benchmark demonstrate that Ferret-UI 2 significantly outperforms Ferret-UI, and also shows strong cross-platform transfer capabilities. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_18967 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Ferret-UI 2: Mastering Universal User Interface Understanding Across Platforms Li, Zhangheng You, Keen Zhang, Haotian Feng, Di Agrawal, Harsh Li, Xiujun Moorthy, Mohana Prasad Sathya Nichols, Jeff Yang, Yinfei Gan, Zhe Computer Vision and Pattern Recognition Computation and Language Machine Learning Building a generalist model for user interface (UI) understanding is challenging due to various foundational issues, such as platform diversity, resolution variation, and data limitation. In this paper, we introduce Ferret-UI 2, a multimodal large language model (MLLM) designed for universal UI understanding across a wide range of platforms, including iPhone, Android, iPad, Webpage, and AppleTV. Building on the foundation of Ferret-UI, Ferret-UI 2 introduces three key innovations: support for multiple platform types, high-resolution perception through adaptive scaling, and advanced task training data generation powered by GPT-4o with set-of-mark visual prompting. These advancements enable Ferret-UI 2 to perform complex, user-centered interactions, making it highly versatile and adaptable for the expanding diversity of platform ecosystems. Extensive empirical experiments on referring, grounding, user-centric advanced tasks (comprising 9 subtasks $\times$ 5 platforms), GUIDE next-action prediction dataset, and GUI-World multi-platform benchmark demonstrate that Ferret-UI 2 significantly outperforms Ferret-UI, and also shows strong cross-platform transfer capabilities. |
| title | Ferret-UI 2: Mastering Universal User Interface Understanding Across Platforms |
| topic | Computer Vision and Pattern Recognition Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2410.18967 |