Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2512.15431 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912777363259392 |
|---|---|
| author | Yan, Haolong Wang, Jia Huang, Xin Shen, Yeqing Meng, Ziyang Fan, Zhimin Tan, Kaijun Gao, Jin Shi, Lieyu Yang, Mi Yang, Shiliang Wang, Zhirui Li, Brian An, Kang Li, Chenyang Lei, Lei Duan, Mengmeng Liang, Danxun Liu, Guodong Cheng, Hang Wu, Hao Dong, Jie Huang, Junhao Chen, Mei Yu, Renjie Li, Shunshan Zhou, Xu Dai, Yiting Deng, Yineng Liang, Yingdan Chen, Zelin Sun, Wen Yan, Chengxu Xu, Chunqin Li, Dong Xiao, Fengqiong Fan, Guanghao Li, Guopeng Peng, Guozhen Li, Hongbing Li, Hang Chen, Hongming Xie, Jingjing Li, Jianyong Zhang, Jingyang Ren, Jiaju Yuan, Jiayu Yin, Jianpeng Cao, Kai Zhao, Liang Tan, Liguo Shi, Liying Ren, Mengqiang Xu, Min Liu, Manjiao Luo, Mao Wan, Mingxin Wang, Na Wu, Nan Wang, Ning Ma, Peiyao Zhang, Qingzhou Wang, Qiao Zeng, Qinlin Gao, Qiong Li, Qiongyao Zhong, Shangwu Gao, Shuli Liu, Shaofan Gao, Shisi Luo, Shuang Liu, Xingbin Liu, Xiaojia Hou, Xiaojie Liu, Xin Feng, Xuanti Cai, Xuedan Wen, Xuan Zhu, Xianwei Liang, Xin Liu, Xin Zhou, Xin Sui, Yifan Zhao, Yingxiu Shi, Yukang Xu, Yunfang Zeng, Yuqing Zhang, Yixun Weng, Zejia Yan, Zhonghao Huang, Zhiguo Wang, Zhuoyu Yan, Zihan Ge, Zheng Li, Jing Zhu, Yibo Jiao, Binxing Zhang, Xiangyu Jiang, Daxin |
| author_facet | Yan, Haolong Wang, Jia Huang, Xin Shen, Yeqing Meng, Ziyang Fan, Zhimin Tan, Kaijun Gao, Jin Shi, Lieyu Yang, Mi Yang, Shiliang Wang, Zhirui Li, Brian An, Kang Li, Chenyang Lei, Lei Duan, Mengmeng Liang, Danxun Liu, Guodong Cheng, Hang Wu, Hao Dong, Jie Huang, Junhao Chen, Mei Yu, Renjie Li, Shunshan Zhou, Xu Dai, Yiting Deng, Yineng Liang, Yingdan Chen, Zelin Sun, Wen Yan, Chengxu Xu, Chunqin Li, Dong Xiao, Fengqiong Fan, Guanghao Li, Guopeng Peng, Guozhen Li, Hongbing Li, Hang Chen, Hongming Xie, Jingjing Li, Jianyong Zhang, Jingyang Ren, Jiaju Yuan, Jiayu Yin, Jianpeng Cao, Kai Zhao, Liang Tan, Liguo Shi, Liying Ren, Mengqiang Xu, Min Liu, Manjiao Luo, Mao Wan, Mingxin Wang, Na Wu, Nan Wang, Ning Ma, Peiyao Zhang, Qingzhou Wang, Qiao Zeng, Qinlin Gao, Qiong Li, Qiongyao Zhong, Shangwu Gao, Shuli Liu, Shaofan Gao, Shisi Luo, Shuang Liu, Xingbin Liu, Xiaojia Hou, Xiaojie Liu, Xin Feng, Xuanti Cai, Xuedan Wen, Xuan Zhu, Xianwei Liang, Xin Liu, Xin Zhou, Xin Sui, Yifan Zhao, Yingxiu Shi, Yukang Xu, Yunfang Zeng, Yuqing Zhang, Yixun Weng, Zejia Yan, Zhonghao Huang, Zhiguo Wang, Zhuoyu Yan, Zihan Ge, Zheng Li, Jing Zhu, Yibo Jiao, Binxing Zhang, Xiangyu Jiang, Daxin |
| contents | Recent advances in multimodal large language models unlock unprecedented opportunities for GUI automation. However, a fundamental challenge remains: how to efficiently acquire high-quality training data while maintaining annotation reliability? We introduce a self-evolving training pipeline powered by the Calibrated Step Reward System, which converts model-generated trajectories into reliable training signals through trajectory-level calibration, achieving >90% annotation accuracy with 10-100x lower cost. Leveraging this pipeline, we introduce Step-GUI, a family of models (4B/8B) that achieves state-of-the-art GUI performance (8B: 80.2% AndroidWorld, 48.5% OSWorld, 62.6% ScreenShot-Pro) while maintaining robust general capabilities. As GUI agent capabilities improve, practical deployment demands standardized interfaces across heterogeneous devices while protecting user privacy. To this end, we propose GUI-MCP, the first Model Context Protocol for GUI automation with hierarchical architecture that combines low-level atomic operations and high-level task delegation to local specialist models, enabling high-privacy execution where sensitive data stays on-device. Finally, to assess whether agents can handle authentic everyday usage, we introduce AndroidDaily, a benchmark grounded in real-world mobile usage patterns with 3146 static actions and 235 end-to-end tasks across high-frequency daily scenarios (8B: static 89.91%, end-to-end 52.50%). Our work advances the development of practical GUI agents and demonstrates strong potential for real-world deployment in everyday digital interactions. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_15431 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Step-GUI Technical Report Yan, Haolong Wang, Jia Huang, Xin Shen, Yeqing Meng, Ziyang Fan, Zhimin Tan, Kaijun Gao, Jin Shi, Lieyu Yang, Mi Yang, Shiliang Wang, Zhirui Li, Brian An, Kang Li, Chenyang Lei, Lei Duan, Mengmeng Liang, Danxun Liu, Guodong Cheng, Hang Wu, Hao Dong, Jie Huang, Junhao Chen, Mei Yu, Renjie Li, Shunshan Zhou, Xu Dai, Yiting Deng, Yineng Liang, Yingdan Chen, Zelin Sun, Wen Yan, Chengxu Xu, Chunqin Li, Dong Xiao, Fengqiong Fan, Guanghao Li, Guopeng Peng, Guozhen Li, Hongbing Li, Hang Chen, Hongming Xie, Jingjing Li, Jianyong Zhang, Jingyang Ren, Jiaju Yuan, Jiayu Yin, Jianpeng Cao, Kai Zhao, Liang Tan, Liguo Shi, Liying Ren, Mengqiang Xu, Min Liu, Manjiao Luo, Mao Wan, Mingxin Wang, Na Wu, Nan Wang, Ning Ma, Peiyao Zhang, Qingzhou Wang, Qiao Zeng, Qinlin Gao, Qiong Li, Qiongyao Zhong, Shangwu Gao, Shuli Liu, Shaofan Gao, Shisi Luo, Shuang Liu, Xingbin Liu, Xiaojia Hou, Xiaojie Liu, Xin Feng, Xuanti Cai, Xuedan Wen, Xuan Zhu, Xianwei Liang, Xin Liu, Xin Zhou, Xin Sui, Yifan Zhao, Yingxiu Shi, Yukang Xu, Yunfang Zeng, Yuqing Zhang, Yixun Weng, Zejia Yan, Zhonghao Huang, Zhiguo Wang, Zhuoyu Yan, Zihan Ge, Zheng Li, Jing Zhu, Yibo Jiao, Binxing Zhang, Xiangyu Jiang, Daxin Computer Vision and Pattern Recognition Recent advances in multimodal large language models unlock unprecedented opportunities for GUI automation. However, a fundamental challenge remains: how to efficiently acquire high-quality training data while maintaining annotation reliability? We introduce a self-evolving training pipeline powered by the Calibrated Step Reward System, which converts model-generated trajectories into reliable training signals through trajectory-level calibration, achieving >90% annotation accuracy with 10-100x lower cost. Leveraging this pipeline, we introduce Step-GUI, a family of models (4B/8B) that achieves state-of-the-art GUI performance (8B: 80.2% AndroidWorld, 48.5% OSWorld, 62.6% ScreenShot-Pro) while maintaining robust general capabilities. As GUI agent capabilities improve, practical deployment demands standardized interfaces across heterogeneous devices while protecting user privacy. To this end, we propose GUI-MCP, the first Model Context Protocol for GUI automation with hierarchical architecture that combines low-level atomic operations and high-level task delegation to local specialist models, enabling high-privacy execution where sensitive data stays on-device. Finally, to assess whether agents can handle authentic everyday usage, we introduce AndroidDaily, a benchmark grounded in real-world mobile usage patterns with 3146 static actions and 235 end-to-end tasks across high-frequency daily scenarios (8B: static 89.91%, end-to-end 52.50%). Our work advances the development of practical GUI agents and demonstrates strong potential for real-world deployment in everyday digital interactions. |
| title | Step-GUI Technical Report |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2512.15431 |