MiMo: Unlocking the Reasoning Potential of Language Model -- From Pretraining to Posttraining
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908394042949632 |
|---|---|
| author | Xiaomi, LLM-Core : Xia, Bingquan Shen, Bowen Cici Zhu, Dawei Zhang, Di Wang, Gang Zhang, Hailin Liu, Huaqiu Xiao, Jiebao Dong, Jinhao Zhao, Liang Li, Peidian Wang, Peng Yu, Shihua Chen, Shimao Wang, Weikun Ma, Wenhan Deng, Xiangwei Huang, Yi Song, Yifan Jiang, Zihan Ye, Bowen Cai, Can He, Chenhong Zhang, Dong Zhang, Duo Wang, Guoan Tian, Hao Zhao, Haochen Qu, Heng Xu, Hongshen Shi, Jun Bao, Kainan Fang, Kai Zhou, Kang Zhou, Kangyang Li, Lei Zhu, Menghang Chen, Nuo Wang, Qiantong Liu, Shaohui Li, Shicheng Gu, Shuhao Ren, Shuhuai Liu, Shuo Deng, Sirui Zhuang, Weiji Lv, Weiwei Yang, Wenyu Zhang, Xin Yong, Xing Zhang, Xing Song, Xingchen Xu, Xinzhe Wang, Xu Yan, Yihan Tu, Yu Tian, Yuanyuan Wang, Yudong Yu, Yue Lin, Zhenru Song, Zhichao Yue, Zihao |
| author_facet | Xiaomi, LLM-Core : Xia, Bingquan Shen, Bowen Cici Zhu, Dawei Zhang, Di Wang, Gang Zhang, Hailin Liu, Huaqiu Xiao, Jiebao Dong, Jinhao Zhao, Liang Li, Peidian Wang, Peng Yu, Shihua Chen, Shimao Wang, Weikun Ma, Wenhan Deng, Xiangwei Huang, Yi Song, Yifan Jiang, Zihan Ye, Bowen Cai, Can He, Chenhong Zhang, Dong Zhang, Duo Wang, Guoan Tian, Hao Zhao, Haochen Qu, Heng Xu, Hongshen Shi, Jun Bao, Kainan Fang, Kai Zhou, Kang Zhou, Kangyang Li, Lei Zhu, Menghang Chen, Nuo Wang, Qiantong Liu, Shaohui Li, Shicheng Gu, Shuhao Ren, Shuhuai Liu, Shuo Deng, Sirui Zhuang, Weiji Lv, Weiwei Yang, Wenyu Zhang, Xin Yong, Xing Zhang, Xing Song, Xingchen Xu, Xinzhe Wang, Xu Yan, Yihan Tu, Yu Tian, Yuanyuan Wang, Yudong Yu, Yue Lin, Zhenru Song, Zhichao Yue, Zihao |
| contents | We present MiMo-7B, a large language model born for reasoning tasks, with optimization across both pre-training and post-training stages. During pre-training, we enhance the data preprocessing pipeline and employ a three-stage data mixing strategy to strengthen the base model's reasoning potential. MiMo-7B-Base is pre-trained on 25 trillion tokens, with additional Multi-Token Prediction objective for enhanced performance and accelerated inference speed. During post-training, we curate a dataset of 130K verifiable mathematics and programming problems for reinforcement learning, integrating a test-difficulty-driven code-reward scheme to alleviate sparse-reward issues and employing strategic data resampling to stabilize training. Extensive evaluations show that MiMo-7B-Base possesses exceptional reasoning potential, outperforming even much larger 32B models. The final RL-tuned model, MiMo-7B-RL, achieves superior performance on mathematics, code and general reasoning tasks, surpassing the performance of OpenAI o1-mini. The model checkpoints are available at https://github.com/xiaomimimo/MiMo. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_07608 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | MiMo: Unlocking the Reasoning Potential of Language Model -- From Pretraining to Posttraining Xiaomi, LLM-Core : Xia, Bingquan Shen, Bowen Cici Zhu, Dawei Zhang, Di Wang, Gang Zhang, Hailin Liu, Huaqiu Xiao, Jiebao Dong, Jinhao Zhao, Liang Li, Peidian Wang, Peng Yu, Shihua Chen, Shimao Wang, Weikun Ma, Wenhan Deng, Xiangwei Huang, Yi Song, Yifan Jiang, Zihan Ye, Bowen Cai, Can He, Chenhong Zhang, Dong Zhang, Duo Wang, Guoan Tian, Hao Zhao, Haochen Qu, Heng Xu, Hongshen Shi, Jun Bao, Kainan Fang, Kai Zhou, Kang Zhou, Kangyang Li, Lei Zhu, Menghang Chen, Nuo Wang, Qiantong Liu, Shaohui Li, Shicheng Gu, Shuhao Ren, Shuhuai Liu, Shuo Deng, Sirui Zhuang, Weiji Lv, Weiwei Yang, Wenyu Zhang, Xin Yong, Xing Zhang, Xing Song, Xingchen Xu, Xinzhe Wang, Xu Yan, Yihan Tu, Yu Tian, Yuanyuan Wang, Yudong Yu, Yue Lin, Zhenru Song, Zhichao Yue, Zihao Computation and Language Artificial Intelligence Machine Learning We present MiMo-7B, a large language model born for reasoning tasks, with optimization across both pre-training and post-training stages. During pre-training, we enhance the data preprocessing pipeline and employ a three-stage data mixing strategy to strengthen the base model's reasoning potential. MiMo-7B-Base is pre-trained on 25 trillion tokens, with additional Multi-Token Prediction objective for enhanced performance and accelerated inference speed. During post-training, we curate a dataset of 130K verifiable mathematics and programming problems for reinforcement learning, integrating a test-difficulty-driven code-reward scheme to alleviate sparse-reward issues and employing strategic data resampling to stabilize training. Extensive evaluations show that MiMo-7B-Base possesses exceptional reasoning potential, outperforming even much larger 32B models. The final RL-tuned model, MiMo-7B-RL, achieves superior performance on mathematics, code and general reasoning tasks, surpassing the performance of OpenAI o1-mini. The model checkpoints are available at https://github.com/xiaomimimo/MiMo. |
| title | MiMo: Unlocking the Reasoning Potential of Language Model -- From Pretraining to Posttraining |
| topic | Computation and Language Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2505.07608 |