MegaScale-MoE: Large-Scale Communication-Efficient Training of Mixture-of-Experts Models in Production
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866912654912651264 |
|---|---|
| author | Jin, Chao Jiang, Ziheng Bai, Zhihao Zhong, Zheng Liu, Juncai Li, Xiang Zheng, Ningxin Wang, Xi Xie, Cong Huang, Qi Heng, Wen Ma, Yiyuan Bao, Wenlei Zheng, Size Peng, Yanghua Lin, Haibin Liu, Xuanzhe Jin, Xin Liu, Xin |
| author_facet | Jin, Chao Jiang, Ziheng Bai, Zhihao Zhong, Zheng Liu, Juncai Li, Xiang Zheng, Ningxin Wang, Xi Xie, Cong Huang, Qi Heng, Wen Ma, Yiyuan Bao, Wenlei Zheng, Size Peng, Yanghua Lin, Haibin Liu, Xuanzhe Jin, Xin Liu, Xin |
| contents | We present MegaScale-MoE, a production system tailored for the efficient training of large-scale mixture-of-experts (MoE) models. MoE emerges as a promising architecture to scale large language models (LLMs) to unprecedented sizes, thereby enhancing model performance. However, existing MoE training systems experience a degradation in training efficiency, exacerbated by the escalating scale of MoE models and the continuous evolution of hardware.
Recognizing the pivotal role of efficient communication in enhancing MoE training, MegaScale-MoE customizes communication-efficient parallelism strategies for attention and FFNs in each MoE layer and adopts a holistic approach to overlap communication with computation at both inter- and intra-operator levels. Additionally, MegaScale-MoE applies communication compression with adjusted communication patterns to lower precision, further improving training efficiency. When training a 352B MoE model on 1,440 NVIDIA Hopper GPUs, MegaScale-MoE achieves a training throughput of 1.41M tokens/s, improving the efficiency by 1.88$\times$ compared to Megatron-LM. We share our operational experience in accelerating MoE training and hope that by offering our insights in system design, this work will motivate future research in MoE systems. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_11432 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | MegaScale-MoE: Large-Scale Communication-Efficient Training of Mixture-of-Experts Models in Production Jin, Chao Jiang, Ziheng Bai, Zhihao Zhong, Zheng Liu, Juncai Li, Xiang Zheng, Ningxin Wang, Xi Xie, Cong Huang, Qi Heng, Wen Ma, Yiyuan Bao, Wenlei Zheng, Size Peng, Yanghua Lin, Haibin Liu, Xuanzhe Jin, Xin Liu, Xin Machine Learning Distributed, Parallel, and Cluster Computing We present MegaScale-MoE, a production system tailored for the efficient training of large-scale mixture-of-experts (MoE) models. MoE emerges as a promising architecture to scale large language models (LLMs) to unprecedented sizes, thereby enhancing model performance. However, existing MoE training systems experience a degradation in training efficiency, exacerbated by the escalating scale of MoE models and the continuous evolution of hardware. Recognizing the pivotal role of efficient communication in enhancing MoE training, MegaScale-MoE customizes communication-efficient parallelism strategies for attention and FFNs in each MoE layer and adopts a holistic approach to overlap communication with computation at both inter- and intra-operator levels. Additionally, MegaScale-MoE applies communication compression with adjusted communication patterns to lower precision, further improving training efficiency. When training a 352B MoE model on 1,440 NVIDIA Hopper GPUs, MegaScale-MoE achieves a training throughput of 1.41M tokens/s, improving the efficiency by 1.88$\times$ compared to Megatron-LM. We share our operational experience in accelerating MoE training and hope that by offering our insights in system design, this work will motivate future research in MoE systems. |
| title | MegaScale-MoE: Large-Scale Communication-Efficient Training of Mixture-of-Experts Models in Production |
| topic | Machine Learning Distributed, Parallel, and Cluster Computing |
| url | https://arxiv.org/abs/2505.11432 |