Copertina

Every FLOP Counts: Scaling a 300B Mixture-of-Experts LING LLM without Premium GPUs

Fuente: arXiv

Salvato in:

Dettagli Bibliografici
Autori principali:	Ling Team, Zeng, Binwei, Huang, Chao, Zhang, Chao, Tian, Changxin, Chen, Cong, Jin, Dingnan, Yu, Feng, Zhu, Feng, Yuan, Feng, Wang, Fakang, Wang, Gangshan, Zhai, Guangyao, Zhang, Haitao, Li, Huizhong, Zhou, Jun, Liu, Jia, Fang, Junpeng, Ou, Junjie, Hu, Jun, Luo, Ji, Zhang, Ji, Liu, Jian, Sha, Jian, Qian, Jianxue, Wu, Jiewei, Zhao, Junping, Li, Jianguo, Feng, Jubao, Di, Jingchao, Xu, Junming, Yao, Jinghua, Xu, Kuan, Du, Kewei, Li, Longfei, Liang, Lei, Yu, Lu, Tang, Li, Ju, Lin, Xu, Peng, Cui, Qing, Liu, Song, Li, Shicheng, Song, Shun, Yan, Song, Cai, Tengwei, Chen, Tianyi, Guo, Ting, Huang, Ting, Feng, Tao, Wu, Tao, Wu, Wei, Zhang, Xiaolu, Yang, Xueming, Zhao, Xin, Hu, Xiaobo, Lin, Xin, Zhao, Yao, Wang, Yilong, Guo, Yongzhen, Wang, Yuanyuan, Yang, Yue, Cao, Yang, Fu, Yuhao, Xiong, Yi, Li, Yanzhe, Li, Zhe, Zhang, Zhiqiang, Liu, Ziqi, Huan, Zhaoxin, Wen, Zujie, Sun, Zhenhang, Du, Zhuoxuan, He, Zhengyu
Natura:	Preprint
Pubblicazione:	2025
Soggetti:	Machine Learning Artificial Intelligence Computation and Language
Accesso online:	Acceder al recurso
Tags:	Aggiungi Tag Nessun Tag, puoi essere il primo ad aggiungerne!!

_version_	1866929750584328192
author	Ling Team Zeng, Binwei Huang, Chao Zhang, Chao Tian, Changxin Chen, Cong Jin, Dingnan Yu, Feng Zhu, Feng Yuan, Feng Wang, Fakang Wang, Gangshan Zhai, Guangyao Zhang, Haitao Li, Huizhong Zhou, Jun Liu, Jia Fang, Junpeng Ou, Junjie Hu, Jun Luo, Ji Zhang, Ji Liu, Jian Sha, Jian Qian, Jianxue Wu, Jiewei Zhao, Junping Li, Jianguo Feng, Jubao Di, Jingchao Xu, Junming Yao, Jinghua Xu, Kuan Du, Kewei Li, Longfei Liang, Lei Yu, Lu Tang, Li Ju, Lin Xu, Peng Cui, Qing Liu, Song Li, Shicheng Song, Shun Yan, Song Cai, Tengwei Chen, Tianyi Guo, Ting Huang, Ting Feng, Tao Wu, Tao Wu, Wei Zhang, Xiaolu Yang, Xueming Zhao, Xin Hu, Xiaobo Lin, Xin Zhao, Yao Wang, Yilong Guo, Yongzhen Wang, Yuanyuan Yang, Yue Cao, Yang Fu, Yuhao Xiong, Yi Li, Yanzhe Li, Zhe Zhang, Zhiqiang Liu, Ziqi Huan, Zhaoxin Wen, Zujie Sun, Zhenhang Du, Zhuoxuan He, Zhengyu
author_facet	Ling Team Zeng, Binwei Huang, Chao Zhang, Chao Tian, Changxin Chen, Cong Jin, Dingnan Yu, Feng Zhu, Feng Yuan, Feng Wang, Fakang Wang, Gangshan Zhai, Guangyao Zhang, Haitao Li, Huizhong Zhou, Jun Liu, Jia Fang, Junpeng Ou, Junjie Hu, Jun Luo, Ji Zhang, Ji Liu, Jian Sha, Jian Qian, Jianxue Wu, Jiewei Zhao, Junping Li, Jianguo Feng, Jubao Di, Jingchao Xu, Junming Yao, Jinghua Xu, Kuan Du, Kewei Li, Longfei Liang, Lei Yu, Lu Tang, Li Ju, Lin Xu, Peng Cui, Qing Liu, Song Li, Shicheng Song, Shun Yan, Song Cai, Tengwei Chen, Tianyi Guo, Ting Huang, Ting Feng, Tao Wu, Tao Wu, Wei Zhang, Xiaolu Yang, Xueming Zhao, Xin Hu, Xiaobo Lin, Xin Zhao, Yao Wang, Yilong Guo, Yongzhen Wang, Yuanyuan Yang, Yue Cao, Yang Fu, Yuhao Xiong, Yi Li, Yanzhe Li, Zhe Zhang, Zhiqiang Liu, Ziqi Huan, Zhaoxin Wen, Zujie Sun, Zhenhang Du, Zhuoxuan He, Zhengyu
contents	In this technical report, we tackle the challenges of training large-scale Mixture of Experts (MoE) models, focusing on overcoming cost inefficiency and resource limitations prevalent in such systems. To address these issues, we present two differently sized MoE large language models (LLMs), namely Ling-Lite and Ling-Plus (referred to as "Bailing" in Chinese, spelled Bǎilíng in Pinyin). Ling-Lite contains 16.8 billion parameters with 2.75 billion activated parameters, while Ling-Plus boasts 290 billion parameters with 28.8 billion activated parameters. Both models exhibit comparable performance to leading industry benchmarks. This report offers actionable insights to improve the efficiency and accessibility of AI development in resource-constrained settings, promoting more scalable and sustainable technologies. Specifically, to reduce training costs for large-scale MoE models, we propose innovative methods for (1) optimization of model architecture and training processes, (2) refinement of training anomaly handling, and (3) enhancement of model evaluation efficiency. Additionally, leveraging high-quality data generated from knowledge graphs, our models demonstrate superior capabilities in tool use compared to other models. Ultimately, our experimental findings demonstrate that a 300B MoE LLM can be effectively trained on lower-performance devices while achieving comparable performance to models of a similar scale, including dense and MoE models. Compared to high-performance devices, utilizing a lower-specification hardware system during the pre-training phase demonstrates significant cost savings, reducing computing costs by approximately 20%. The models can be accessed at https://huggingface.co/inclusionAI.
format	Preprint
id	arxiv_https___arxiv_org_abs_2503_05139
institution	arXiv
publishDate	2025
record_format	arxiv
spellingShingle	Every FLOP Counts: Scaling a 300B Mixture-of-Experts LING LLM without Premium GPUs Ling Team Zeng, Binwei Huang, Chao Zhang, Chao Tian, Changxin Chen, Cong Jin, Dingnan Yu, Feng Zhu, Feng Yuan, Feng Wang, Fakang Wang, Gangshan Zhai, Guangyao Zhang, Haitao Li, Huizhong Zhou, Jun Liu, Jia Fang, Junpeng Ou, Junjie Hu, Jun Luo, Ji Zhang, Ji Liu, Jian Sha, Jian Qian, Jianxue Wu, Jiewei Zhao, Junping Li, Jianguo Feng, Jubao Di, Jingchao Xu, Junming Yao, Jinghua Xu, Kuan Du, Kewei Li, Longfei Liang, Lei Yu, Lu Tang, Li Ju, Lin Xu, Peng Cui, Qing Liu, Song Li, Shicheng Song, Shun Yan, Song Cai, Tengwei Chen, Tianyi Guo, Ting Huang, Ting Feng, Tao Wu, Tao Wu, Wei Zhang, Xiaolu Yang, Xueming Zhao, Xin Hu, Xiaobo Lin, Xin Zhao, Yao Wang, Yilong Guo, Yongzhen Wang, Yuanyuan Yang, Yue Cao, Yang Fu, Yuhao Xiong, Yi Li, Yanzhe Li, Zhe Zhang, Zhiqiang Liu, Ziqi Huan, Zhaoxin Wen, Zujie Sun, Zhenhang Du, Zhuoxuan He, Zhengyu Machine Learning Artificial Intelligence Computation and Language In this technical report, we tackle the challenges of training large-scale Mixture of Experts (MoE) models, focusing on overcoming cost inefficiency and resource limitations prevalent in such systems. To address these issues, we present two differently sized MoE large language models (LLMs), namely Ling-Lite and Ling-Plus (referred to as "Bailing" in Chinese, spelled Bǎilíng in Pinyin). Ling-Lite contains 16.8 billion parameters with 2.75 billion activated parameters, while Ling-Plus boasts 290 billion parameters with 28.8 billion activated parameters. Both models exhibit comparable performance to leading industry benchmarks. This report offers actionable insights to improve the efficiency and accessibility of AI development in resource-constrained settings, promoting more scalable and sustainable technologies. Specifically, to reduce training costs for large-scale MoE models, we propose innovative methods for (1) optimization of model architecture and training processes, (2) refinement of training anomaly handling, and (3) enhancement of model evaluation efficiency. Additionally, leveraging high-quality data generated from knowledge graphs, our models demonstrate superior capabilities in tool use compared to other models. Ultimately, our experimental findings demonstrate that a 300B MoE LLM can be effectively trained on lower-performance devices while achieving comparable performance to models of a similar scale, including dense and MoE models. Compared to high-performance devices, utilizing a lower-specification hardware system during the pre-training phase demonstrates significant cost savings, reducing computing costs by approximately 20%. The models can be accessed at https://huggingface.co/inclusionAI.
title	Every FLOP Counts: Scaling a 300B Mixture-of-Experts LING LLM without Premium GPUs
topic	Machine Learning Artificial Intelligence Computation and Language
url	https://arxiv.org/abs/2503.05139