AquilaMoE: Efficient Training for MoE Models with Scale-Up and Scale-Out Strategies
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866910564683350016 |
|---|---|
| author | Zhang, Bo-Wen Wang, Liangdong Yuan, Ye Li, Jijie Gu, Shuhao Zhao, Mengdi Wu, Xinya Liu, Guang Wu, Chengwei Zhao, Hanyu Du, Li Ju, Yiming Ma, Quanyue Ao, Yulong Zhao, Yingli Zhu, Songhe Cao, Zhou Liang, Dong Lin, Yonghua Zhang, Ming Wang, Shunfei Zhou, Yanxin Ye, Min Chen, Xuekai Yu, Xinyang Huang, Xiangjun Yang, Jian |
| author_facet | Zhang, Bo-Wen Wang, Liangdong Yuan, Ye Li, Jijie Gu, Shuhao Zhao, Mengdi Wu, Xinya Liu, Guang Wu, Chengwei Zhao, Hanyu Du, Li Ju, Yiming Ma, Quanyue Ao, Yulong Zhao, Yingli Zhu, Songhe Cao, Zhou Liang, Dong Lin, Yonghua Zhang, Ming Wang, Shunfei Zhou, Yanxin Ye, Min Chen, Xuekai Yu, Xinyang Huang, Xiangjun Yang, Jian |
| contents | In recent years, with the rapid application of large language models across various fields, the scale of these models has gradually increased, and the resources required for their pre-training have grown exponentially. Training an LLM from scratch will cost a lot of computation resources while scaling up from a smaller model is a more efficient approach and has thus attracted significant attention. In this paper, we present AquilaMoE, a cutting-edge bilingual 8*16B Mixture of Experts (MoE) language model that has 8 experts with 16 billion parameters each and is developed using an innovative training methodology called EfficientScale. This approach optimizes performance while minimizing data requirements through a two-stage process. The first stage, termed Scale-Up, initializes the larger model with weights from a pre-trained smaller model, enabling substantial knowledge transfer and continuous pretraining with significantly less data. The second stage, Scale-Out, uses a pre-trained dense model to initialize the MoE experts, further enhancing knowledge transfer and performance. Extensive validation experiments on 1.8B and 7B models compared various initialization schemes, achieving models that maintain and reduce loss during continuous pretraining. Utilizing the optimal scheme, we successfully trained a 16B model and subsequently the 8*16B AquilaMoE model, demonstrating significant improvements in performance and training efficiency. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_06567 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | AquilaMoE: Efficient Training for MoE Models with Scale-Up and Scale-Out Strategies Zhang, Bo-Wen Wang, Liangdong Yuan, Ye Li, Jijie Gu, Shuhao Zhao, Mengdi Wu, Xinya Liu, Guang Wu, Chengwei Zhao, Hanyu Du, Li Ju, Yiming Ma, Quanyue Ao, Yulong Zhao, Yingli Zhu, Songhe Cao, Zhou Liang, Dong Lin, Yonghua Zhang, Ming Wang, Shunfei Zhou, Yanxin Ye, Min Chen, Xuekai Yu, Xinyang Huang, Xiangjun Yang, Jian Computation and Language Artificial Intelligence In recent years, with the rapid application of large language models across various fields, the scale of these models has gradually increased, and the resources required for their pre-training have grown exponentially. Training an LLM from scratch will cost a lot of computation resources while scaling up from a smaller model is a more efficient approach and has thus attracted significant attention. In this paper, we present AquilaMoE, a cutting-edge bilingual 8*16B Mixture of Experts (MoE) language model that has 8 experts with 16 billion parameters each and is developed using an innovative training methodology called EfficientScale. This approach optimizes performance while minimizing data requirements through a two-stage process. The first stage, termed Scale-Up, initializes the larger model with weights from a pre-trained smaller model, enabling substantial knowledge transfer and continuous pretraining with significantly less data. The second stage, Scale-Out, uses a pre-trained dense model to initialize the MoE experts, further enhancing knowledge transfer and performance. Extensive validation experiments on 1.8B and 7B models compared various initialization schemes, achieving models that maintain and reduce loss during continuous pretraining. Utilizing the optimal scheme, we successfully trained a 16B model and subsequently the 8*16B AquilaMoE model, demonstrating significant improvements in performance and training efficiency. |
| title | AquilaMoE: Efficient Training for MoE Models with Scale-Up and Scale-Out Strategies |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2408.06567 |