Scalable Training of Mixture-of-Experts Models with Megatron Core
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914380565708800 |
|---|---|
| author | Yan, Zijie Bai, Hongxiao Yao, Xin Liu, Dennis Liu, Tong Liu, Hongbin Li, Pingtian Wu, Evan Fan, Shiqing Tao, Li Zhang, Robin Wang, Yuzhong Xu, Shifang Chang, Jack Chen, Xuwen Li, Kunlun Bai, Yan Deng, Gao Zheng, Nan Korthikanti, Vijay Anand Khattar, Abhinav He, Ethan Govande, Soham Lym, Sangkug Zhu, Zhongbo Zhang, Qi Yuan, Haochen Ren, Xiaowei Fu, Deyu Ma, Tailai Zhang, Shunkang Shao, Jiang Wang, Ray Rengasamy, Vasudevan Garg, Rachit Bhavani, Santosh Li, Xipeng Zhou, Chandler Wu, David Wei, Yingcan Aithal, Ashwath Andersch, Michael Shoeybi, Mohammad Yao, Jiajie Yang, June |
| author_facet | Yan, Zijie Bai, Hongxiao Yao, Xin Liu, Dennis Liu, Tong Liu, Hongbin Li, Pingtian Wu, Evan Fan, Shiqing Tao, Li Zhang, Robin Wang, Yuzhong Xu, Shifang Chang, Jack Chen, Xuwen Li, Kunlun Bai, Yan Deng, Gao Zheng, Nan Korthikanti, Vijay Anand Khattar, Abhinav He, Ethan Govande, Soham Lym, Sangkug Zhu, Zhongbo Zhang, Qi Yuan, Haochen Ren, Xiaowei Fu, Deyu Ma, Tailai Zhang, Shunkang Shao, Jiang Wang, Ray Rengasamy, Vasudevan Garg, Rachit Bhavani, Santosh Li, Xipeng Zhou, Chandler Wu, David Wei, Yingcan Aithal, Ashwath Andersch, Michael Shoeybi, Mohammad Yao, Jiajie Yang, June |
| contents | Scaling Mixture-of-Experts (MoE) training introduces systems challenges absent in dense models. Because each token activates only a subset of experts, this sparsity allows total parameters to grow much faster than per-token computation, creating coupled constraints across memory, communication, and computation. Optimizing one dimension often shifts pressure to another, demanding co-design across the full system stack.
We address these challenges for MoE training through integrated optimizations spanning memory (fine-grained recomputation, offloading, etc.), communication (optimized dispatchers, overlapping, etc.), and computation (Grouped GEMM, fusions, CUDA Graphs, etc.). The framework also provides Parallel Folding for flexible multi-dimensional parallelism, low-precision training support for FP8 and NVFP4, and efficient long-context training. On NVIDIA GB300 and GB200, it achieves 1,233/1,048 TFLOPS/GPU for DeepSeek-V3-685B and 974/919 TFLOPS/GPU for Qwen3-235B. As a performant, scalable, and production-ready open-source solution, it has been used across academia and industry for training MoE models ranging from billions to trillions of parameters on clusters scaling up to thousands of GPUs.
This report explains how these techniques work, their trade-offs, and their interactions at the systems level, providing practical guidance for scaling MoE models with Megatron Core. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_07685 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Scalable Training of Mixture-of-Experts Models with Megatron Core Yan, Zijie Bai, Hongxiao Yao, Xin Liu, Dennis Liu, Tong Liu, Hongbin Li, Pingtian Wu, Evan Fan, Shiqing Tao, Li Zhang, Robin Wang, Yuzhong Xu, Shifang Chang, Jack Chen, Xuwen Li, Kunlun Bai, Yan Deng, Gao Zheng, Nan Korthikanti, Vijay Anand Khattar, Abhinav He, Ethan Govande, Soham Lym, Sangkug Zhu, Zhongbo Zhang, Qi Yuan, Haochen Ren, Xiaowei Fu, Deyu Ma, Tailai Zhang, Shunkang Shao, Jiang Wang, Ray Rengasamy, Vasudevan Garg, Rachit Bhavani, Santosh Li, Xipeng Zhou, Chandler Wu, David Wei, Yingcan Aithal, Ashwath Andersch, Michael Shoeybi, Mohammad Yao, Jiajie Yang, June Distributed, Parallel, and Cluster Computing Computation and Language Machine Learning Scaling Mixture-of-Experts (MoE) training introduces systems challenges absent in dense models. Because each token activates only a subset of experts, this sparsity allows total parameters to grow much faster than per-token computation, creating coupled constraints across memory, communication, and computation. Optimizing one dimension often shifts pressure to another, demanding co-design across the full system stack. We address these challenges for MoE training through integrated optimizations spanning memory (fine-grained recomputation, offloading, etc.), communication (optimized dispatchers, overlapping, etc.), and computation (Grouped GEMM, fusions, CUDA Graphs, etc.). The framework also provides Parallel Folding for flexible multi-dimensional parallelism, low-precision training support for FP8 and NVFP4, and efficient long-context training. On NVIDIA GB300 and GB200, it achieves 1,233/1,048 TFLOPS/GPU for DeepSeek-V3-685B and 974/919 TFLOPS/GPU for Qwen3-235B. As a performant, scalable, and production-ready open-source solution, it has been used across academia and industry for training MoE models ranging from billions to trillions of parameters on clusters scaling up to thousands of GPUs. This report explains how these techniques work, their trade-offs, and their interactions at the systems level, providing practical guidance for scaling MoE models with Megatron Core. |
| title | Scalable Training of Mixture-of-Experts Models with Megatron Core |
| topic | Distributed, Parallel, and Cluster Computing Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2603.07685 |