FlowMoE: A Scalable Pipeline Scheduling Framework for Distributed Mixture-of-Experts Training
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866918155792678912 |
|---|---|
| author | Gao, Yunqi Hu, Bing Mashhadi, Mahdi Boloursaz Jin, A-Long Zhang, Yanfeng Xiao, Pei Tafazolli, Rahim Debbah, Merouane |
| author_facet | Gao, Yunqi Hu, Bing Mashhadi, Mahdi Boloursaz Jin, A-Long Zhang, Yanfeng Xiao, Pei Tafazolli, Rahim Debbah, Merouane |
| contents | The parameter size of modern large language models (LLMs) can be scaled up via the sparsely-activated Mixture-of-Experts (MoE) technique to avoid excessive increase of the computational costs. To further improve training efficiency, pipelining computation and communication has become a promising solution for distributed MoE training. However, existing work primarily focuses on scheduling tasks within the MoE layer, such as expert computing and all-to-all (A2A) communication, while neglecting other key operations including multi-head attention (MHA) computing, gating, and all-reduce communication. In this paper, we propose FlowMoE, a scalable framework for scheduling multi-type task pipelines. First, FlowMoE constructs a unified pipeline to consistently scheduling MHA computing, gating, expert computing, and A2A communication. Second, FlowMoE introduces a tensor chunk-based priority scheduling mechanism to overlap the all-reduce communication with all computing tasks. We implement FlowMoE as an adaptive and generic framework atop PyTorch. Extensive experiments with 675 typical MoE layers and four real-world MoE models across two GPU clusters demonstrate that our proposed FlowMoE framework outperforms state-of-the-art MoE training frameworks, reducing training time by 13%-57%, energy consumption by 10%-39%, and memory usage by 7%-32%. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_00207 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | FlowMoE: A Scalable Pipeline Scheduling Framework for Distributed Mixture-of-Experts Training Gao, Yunqi Hu, Bing Mashhadi, Mahdi Boloursaz Jin, A-Long Zhang, Yanfeng Xiao, Pei Tafazolli, Rahim Debbah, Merouane Distributed, Parallel, and Cluster Computing The parameter size of modern large language models (LLMs) can be scaled up via the sparsely-activated Mixture-of-Experts (MoE) technique to avoid excessive increase of the computational costs. To further improve training efficiency, pipelining computation and communication has become a promising solution for distributed MoE training. However, existing work primarily focuses on scheduling tasks within the MoE layer, such as expert computing and all-to-all (A2A) communication, while neglecting other key operations including multi-head attention (MHA) computing, gating, and all-reduce communication. In this paper, we propose FlowMoE, a scalable framework for scheduling multi-type task pipelines. First, FlowMoE constructs a unified pipeline to consistently scheduling MHA computing, gating, expert computing, and A2A communication. Second, FlowMoE introduces a tensor chunk-based priority scheduling mechanism to overlap the all-reduce communication with all computing tasks. We implement FlowMoE as an adaptive and generic framework atop PyTorch. Extensive experiments with 675 typical MoE layers and four real-world MoE models across two GPU clusters demonstrate that our proposed FlowMoE framework outperforms state-of-the-art MoE training frameworks, reducing training time by 13%-57%, energy consumption by 10%-39%, and memory usage by 7%-32%. |
| title | FlowMoE: A Scalable Pipeline Scheduling Framework for Distributed Mixture-of-Experts Training |
| topic | Distributed, Parallel, and Cluster Computing |
| url | https://arxiv.org/abs/2510.00207 |