Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2509.17336 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908621985546240 |
|---|---|
| author | Fu, Tianyu Su, Anyang Zhao, Chenxu Wang, Hanning Wu, Minghui Yu, Zhe Hu, Fei Shi, Mingjia Dong, Wei Wang, Jiayao Chen, Yuyang Yu, Ruiyang Peng, Siran Li, Menglin Huang, Nan Wei, Haitian Yu, Jiawei Xin, Yi Zhao, Xilin Gu, Kai Jiang, Ping Zhou, Sifan Wang, Shuo |
| author_facet | Fu, Tianyu Su, Anyang Zhao, Chenxu Wang, Hanning Wu, Minghui Yu, Zhe Hu, Fei Shi, Mingjia Dong, Wei Wang, Jiayao Chen, Yuyang Yu, Ruiyang Peng, Siran Li, Menglin Huang, Nan Wei, Haitian Yu, Jiawei Xin, Yi Zhao, Xilin Gu, Kai Jiang, Ping Zhou, Sifan Wang, Shuo |
| contents | Graphical user interfaces (GUIs) are the primary medium for human-computer interaction, yet automating GUI interactions remains challenging due to the complexity of visual elements, dynamic environments, and the need for multi-step reasoning. Existing methods based on vision-language models (VLMs) often suffer from limited resolution, domain mismatch, and insufficient sequential decisionmaking capability. To address these issues, we propose Mano, a robust GUI agent built upon a multi-modal foundation model pre-trained on extensive web and computer system data. Our approach integrates a novel simulated environment for high-fidelity data generation, a three-stage training pipeline (supervised fine-tuning, offline reinforcement learning, and online reinforcement learning), and a verification module for error recovery. Mano demonstrates state-of-the-art performance on multiple GUI benchmarks, including Mind2Web and OSWorld, achieving significant improvements in success rate and operational accuracy. Our work provides new insights into the effective integration of reinforcement learning with VLMs for practical GUI agent deployment, highlighting the importance of domain-specific data, iterative training, and holistic reward design. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_17336 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Mano Technical Report Fu, Tianyu Su, Anyang Zhao, Chenxu Wang, Hanning Wu, Minghui Yu, Zhe Hu, Fei Shi, Mingjia Dong, Wei Wang, Jiayao Chen, Yuyang Yu, Ruiyang Peng, Siran Li, Menglin Huang, Nan Wei, Haitian Yu, Jiawei Xin, Yi Zhao, Xilin Gu, Kai Jiang, Ping Zhou, Sifan Wang, Shuo Multimedia Computation and Language Computer Vision and Pattern Recognition Graphical user interfaces (GUIs) are the primary medium for human-computer interaction, yet automating GUI interactions remains challenging due to the complexity of visual elements, dynamic environments, and the need for multi-step reasoning. Existing methods based on vision-language models (VLMs) often suffer from limited resolution, domain mismatch, and insufficient sequential decisionmaking capability. To address these issues, we propose Mano, a robust GUI agent built upon a multi-modal foundation model pre-trained on extensive web and computer system data. Our approach integrates a novel simulated environment for high-fidelity data generation, a three-stage training pipeline (supervised fine-tuning, offline reinforcement learning, and online reinforcement learning), and a verification module for error recovery. Mano demonstrates state-of-the-art performance on multiple GUI benchmarks, including Mind2Web and OSWorld, achieving significant improvements in success rate and operational accuracy. Our work provides new insights into the effective integration of reinforcement learning with VLMs for practical GUI agent deployment, highlighting the importance of domain-specific data, iterative training, and holistic reward design. |
| title | Mano Technical Report |
| topic | Multimedia Computation and Language Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2509.17336 |