Salvato in:
| Autori principali: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | https://arxiv.org/abs/2501.08313 |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866929676043157504 |
|---|---|
| author | MiniMax Li, Aonian Gong, Bangwei Yang, Bo Shan, Boji Liu, Chang Zhu, Cheng Zhang, Chunhao Guo, Congchao Chen, Da Li, Dong Jiao, Enwei Li, Gengxin Zhang, Guojun Sun, Haohai Dong, Houze Zhu, Jiadai Zhuang, Jiaqi Song, Jiayuan Zhu, Jin Han, Jingtao Li, Jingyang Xie, Junbin Xu, Junhao Yan, Junjie Zhang, Kaishun Xiao, Kecheng Kang, Kexi Han, Le Wang, Leyang Yu, Lianfei Feng, Liheng Zheng, Lin Chai, Linbo Xing, Long Ju, Meizhi Chi, Mingyuan Zhang, Mozhi Huang, Peikai Niu, Pengcheng Li, Pengfei Zhao, Pengyu Yang, Qi Xu, Qidi Wang, Qiexiang Wang, Qin Li, Qiuhui Leng, Ruitao Shi, Shengmin Yu, Shuqi Li, Sichen Zhu, Songquan Huang, Tao Liang, Tianrun Sun, Weigao Sun, Weixuan Cheng, Weiyu Li, Wenkai Song, Xiangjun Su, Xiao Han, Xiaodong Zhang, Xinjie Hou, Xinzhu Min, Xu Zou, Xun Shen, Xuyang Gong, Yan Zhu, Yingjie Zhou, Yipeng Zhong, Yiran Hu, Yongyi Fan, Yuanxiang Yu, Yue Yang, Yufeng Li, Yuhao Huang, Yunan Li, Yunji Huang, Yunpeng Xu, Yunzhi Mao, Yuxin Li, Zehan Li, Zekang Tao, Zewei Ying, Zewen Cong, Zhaoyang Qin, Zhen Fan, Zhenhua Yu, Zhihang Jiang, Zhuo Wu, Zijia |
| author_facet | MiniMax Li, Aonian Gong, Bangwei Yang, Bo Shan, Boji Liu, Chang Zhu, Cheng Zhang, Chunhao Guo, Congchao Chen, Da Li, Dong Jiao, Enwei Li, Gengxin Zhang, Guojun Sun, Haohai Dong, Houze Zhu, Jiadai Zhuang, Jiaqi Song, Jiayuan Zhu, Jin Han, Jingtao Li, Jingyang Xie, Junbin Xu, Junhao Yan, Junjie Zhang, Kaishun Xiao, Kecheng Kang, Kexi Han, Le Wang, Leyang Yu, Lianfei Feng, Liheng Zheng, Lin Chai, Linbo Xing, Long Ju, Meizhi Chi, Mingyuan Zhang, Mozhi Huang, Peikai Niu, Pengcheng Li, Pengfei Zhao, Pengyu Yang, Qi Xu, Qidi Wang, Qiexiang Wang, Qin Li, Qiuhui Leng, Ruitao Shi, Shengmin Yu, Shuqi Li, Sichen Zhu, Songquan Huang, Tao Liang, Tianrun Sun, Weigao Sun, Weixuan Cheng, Weiyu Li, Wenkai Song, Xiangjun Su, Xiao Han, Xiaodong Zhang, Xinjie Hou, Xinzhu Min, Xu Zou, Xun Shen, Xuyang Gong, Yan Zhu, Yingjie Zhou, Yipeng Zhong, Yiran Hu, Yongyi Fan, Yuanxiang Yu, Yue Yang, Yufeng Li, Yuhao Huang, Yunan Li, Yunji Huang, Yunpeng Xu, Yunzhi Mao, Yuxin Li, Zehan Li, Zekang Tao, Zewei Ying, Zewen Cong, Zhaoyang Qin, Zhen Fan, Zhenhua Yu, Zhihang Jiang, Zhuo Wu, Zijia |
| contents | We introduce MiniMax-01 series, including MiniMax-Text-01 and MiniMax-VL-01, which are comparable to top-tier models while offering superior capabilities in processing longer contexts. The core lies in lightning attention and its efficient scaling. To maximize computational capacity, we integrate it with Mixture of Experts (MoE), creating a model with 32 experts and 456 billion total parameters, of which 45.9 billion are activated for each token. We develop an optimized parallel strategy and highly efficient computation-communication overlap techniques for MoE and lightning attention. This approach enables us to conduct efficient training and inference on models with hundreds of billions of parameters across contexts spanning millions of tokens. The context window of MiniMax-Text-01 can reach up to 1 million tokens during training and extrapolate to 4 million tokens during inference at an affordable cost. Our vision-language model, MiniMax-VL-01 is built through continued training with 512 billion vision-language tokens. Experiments on both standard and in-house benchmarks show that our models match the performance of state-of-the-art models like GPT-4o and Claude-3.5-Sonnet while offering 20-32 times longer context window. We publicly release MiniMax-01 at https://github.com/MiniMax-AI. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_08313 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | MiniMax-01: Scaling Foundation Models with Lightning Attention MiniMax Li, Aonian Gong, Bangwei Yang, Bo Shan, Boji Liu, Chang Zhu, Cheng Zhang, Chunhao Guo, Congchao Chen, Da Li, Dong Jiao, Enwei Li, Gengxin Zhang, Guojun Sun, Haohai Dong, Houze Zhu, Jiadai Zhuang, Jiaqi Song, Jiayuan Zhu, Jin Han, Jingtao Li, Jingyang Xie, Junbin Xu, Junhao Yan, Junjie Zhang, Kaishun Xiao, Kecheng Kang, Kexi Han, Le Wang, Leyang Yu, Lianfei Feng, Liheng Zheng, Lin Chai, Linbo Xing, Long Ju, Meizhi Chi, Mingyuan Zhang, Mozhi Huang, Peikai Niu, Pengcheng Li, Pengfei Zhao, Pengyu Yang, Qi Xu, Qidi Wang, Qiexiang Wang, Qin Li, Qiuhui Leng, Ruitao Shi, Shengmin Yu, Shuqi Li, Sichen Zhu, Songquan Huang, Tao Liang, Tianrun Sun, Weigao Sun, Weixuan Cheng, Weiyu Li, Wenkai Song, Xiangjun Su, Xiao Han, Xiaodong Zhang, Xinjie Hou, Xinzhu Min, Xu Zou, Xun Shen, Xuyang Gong, Yan Zhu, Yingjie Zhou, Yipeng Zhong, Yiran Hu, Yongyi Fan, Yuanxiang Yu, Yue Yang, Yufeng Li, Yuhao Huang, Yunan Li, Yunji Huang, Yunpeng Xu, Yunzhi Mao, Yuxin Li, Zehan Li, Zekang Tao, Zewei Ying, Zewen Cong, Zhaoyang Qin, Zhen Fan, Zhenhua Yu, Zhihang Jiang, Zhuo Wu, Zijia Computation and Language Computer Vision and Pattern Recognition We introduce MiniMax-01 series, including MiniMax-Text-01 and MiniMax-VL-01, which are comparable to top-tier models while offering superior capabilities in processing longer contexts. The core lies in lightning attention and its efficient scaling. To maximize computational capacity, we integrate it with Mixture of Experts (MoE), creating a model with 32 experts and 456 billion total parameters, of which 45.9 billion are activated for each token. We develop an optimized parallel strategy and highly efficient computation-communication overlap techniques for MoE and lightning attention. This approach enables us to conduct efficient training and inference on models with hundreds of billions of parameters across contexts spanning millions of tokens. The context window of MiniMax-Text-01 can reach up to 1 million tokens during training and extrapolate to 4 million tokens during inference at an affordable cost. Our vision-language model, MiniMax-VL-01 is built through continued training with 512 billion vision-language tokens. Experiments on both standard and in-house benchmarks show that our models match the performance of state-of-the-art models like GPT-4o and Claude-3.5-Sonnet while offering 20-32 times longer context window. We publicly release MiniMax-01 at https://github.com/MiniMax-AI. |
| title | MiniMax-01: Scaling Foundation Models with Lightning Attention |
| topic | Computation and Language Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2501.08313 |