Ferret-UI Lite: Lessons from Building Small On-Device GUI Agents
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916980387217408 |
|---|---|
| author | Yang, Zhen Dou, Zi-Yi Feng, Di Huang, Forrest Nguyen, Anh You, Keen Attia, Omar Yang, Yuhao Feng, Michael Zhang, Haotian Ramrakhya, Ram Jia, Chao Nichols, Jeffrey Toshev, Alexander Yang, Yinfei Gan, Zhe |
| author_facet | Yang, Zhen Dou, Zi-Yi Feng, Di Huang, Forrest Nguyen, Anh You, Keen Attia, Omar Yang, Yuhao Feng, Michael Zhang, Haotian Ramrakhya, Ram Jia, Chao Nichols, Jeffrey Toshev, Alexander Yang, Yinfei Gan, Zhe |
| contents | Developing autonomous agents that effectively interact with Graphic User Interfaces (GUIs) remains a challenging open problem, especially for small on-device models. In this paper, we present Ferret-UI Lite, a compact, end-to-end GUI agent that operates across diverse platforms, including mobile, web, and desktop. Utilizing techniques optimized for developing small models, we build our 3B Ferret-UI Lite agent through curating a diverse GUI data mixture from real and synthetic sources, strengthening inference-time performance through chain-of-thought reasoning and visual tool-use, and reinforcement learning with designed rewards. Ferret-UI Lite achieves competitive performance with other small-scale GUI agents. In GUI grounding, Ferret-UI Lite attains scores of $91.6\%$, $53.3\%$, and $61.2\%$ on the ScreenSpot-V2, ScreenSpot-Pro, and OSWorld-G benchmarks, respectively. For GUI navigation, Ferret-UI Lite achieves success rates of $28.0\%$ on AndroidWorld and $19.8\%$ on OSWorld. We share our methods and lessons learned from developing compact, on-device GUI agents. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_26539 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Ferret-UI Lite: Lessons from Building Small On-Device GUI Agents Yang, Zhen Dou, Zi-Yi Feng, Di Huang, Forrest Nguyen, Anh You, Keen Attia, Omar Yang, Yuhao Feng, Michael Zhang, Haotian Ramrakhya, Ram Jia, Chao Nichols, Jeffrey Toshev, Alexander Yang, Yinfei Gan, Zhe Computer Vision and Pattern Recognition Computation and Language Machine Learning Developing autonomous agents that effectively interact with Graphic User Interfaces (GUIs) remains a challenging open problem, especially for small on-device models. In this paper, we present Ferret-UI Lite, a compact, end-to-end GUI agent that operates across diverse platforms, including mobile, web, and desktop. Utilizing techniques optimized for developing small models, we build our 3B Ferret-UI Lite agent through curating a diverse GUI data mixture from real and synthetic sources, strengthening inference-time performance through chain-of-thought reasoning and visual tool-use, and reinforcement learning with designed rewards. Ferret-UI Lite achieves competitive performance with other small-scale GUI agents. In GUI grounding, Ferret-UI Lite attains scores of $91.6\%$, $53.3\%$, and $61.2\%$ on the ScreenSpot-V2, ScreenSpot-Pro, and OSWorld-G benchmarks, respectively. For GUI navigation, Ferret-UI Lite achieves success rates of $28.0\%$ on AndroidWorld and $19.8\%$ on OSWorld. We share our methods and lessons learned from developing compact, on-device GUI agents. |
| title | Ferret-UI Lite: Lessons from Building Small On-Device GUI Agents |
| topic | Computer Vision and Pattern Recognition Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2509.26539 |