MegaScale-Infer: Serving Mixture-of-Experts at Scale with Disaggregated Expert Parallelism
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866912501817409536 |
|---|---|
| author | Zhu, Ruidong Jiang, Ziheng Jin, Chao Wu, Peng Stuardo, Cesar A. Wang, Dongyang Zhang, Xinlei Zhou, Huaping Wei, Haoran Cheng, Yang Xiao, Jianzhe Zhang, Xinyi Liu, Lingjun Lin, Haibin Chang, Li-Wen Ye, Jianxi Yu, Xiao Liu, Xuanzhe Jin, Xin Liu, Xin |
| author_facet | Zhu, Ruidong Jiang, Ziheng Jin, Chao Wu, Peng Stuardo, Cesar A. Wang, Dongyang Zhang, Xinlei Zhou, Huaping Wei, Haoran Cheng, Yang Xiao, Jianzhe Zhang, Xinyi Liu, Lingjun Lin, Haibin Chang, Li-Wen Ye, Jianxi Yu, Xiao Liu, Xuanzhe Jin, Xin Liu, Xin |
| contents | Mixture-of-Experts (MoE) showcases tremendous potential to scale large language models (LLMs) with enhanced performance and reduced computational complexity. However, its sparsely activated architecture shifts feed-forward networks (FFNs) from being compute-intensive to memory-intensive during inference, leading to substantially lower GPU utilization and increased operational costs.
We present MegaScale-Infer, an efficient and cost-effective system for serving large-scale MoE models. MegaScale-Infer disaggregates attention and FFN modules within each model layer, enabling independent scaling, tailored parallelism strategies, and heterogeneous deployment for both modules. To fully exploit disaggregation in the presence of MoE's sparsity, MegaScale-Infer introduces ping-pong pipeline parallelism, which partitions a request batch into micro-batches and shuttles them between attention and FFNs for inference. Combined with distinct model parallelism for each module, MegaScale-Infer effectively hides communication overhead and maximizes GPU utilization. To adapt to disaggregated attention and FFN modules and minimize data transmission overhead (e.g., token dispatch), MegaScale-Infer provides a high-performance M2N communication library that eliminates unnecessary GPU-to-CPU data copies, group initialization overhead, and GPU synchronization. Experimental results indicate that MegaScale-Infer achieves up to 1.90x higher per-GPU throughput than state-of-the-art solutions. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_02263 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | MegaScale-Infer: Serving Mixture-of-Experts at Scale with Disaggregated Expert Parallelism Zhu, Ruidong Jiang, Ziheng Jin, Chao Wu, Peng Stuardo, Cesar A. Wang, Dongyang Zhang, Xinlei Zhou, Huaping Wei, Haoran Cheng, Yang Xiao, Jianzhe Zhang, Xinyi Liu, Lingjun Lin, Haibin Chang, Li-Wen Ye, Jianxi Yu, Xiao Liu, Xuanzhe Jin, Xin Liu, Xin Distributed, Parallel, and Cluster Computing Machine Learning Mixture-of-Experts (MoE) showcases tremendous potential to scale large language models (LLMs) with enhanced performance and reduced computational complexity. However, its sparsely activated architecture shifts feed-forward networks (FFNs) from being compute-intensive to memory-intensive during inference, leading to substantially lower GPU utilization and increased operational costs. We present MegaScale-Infer, an efficient and cost-effective system for serving large-scale MoE models. MegaScale-Infer disaggregates attention and FFN modules within each model layer, enabling independent scaling, tailored parallelism strategies, and heterogeneous deployment for both modules. To fully exploit disaggregation in the presence of MoE's sparsity, MegaScale-Infer introduces ping-pong pipeline parallelism, which partitions a request batch into micro-batches and shuttles them between attention and FFNs for inference. Combined with distinct model parallelism for each module, MegaScale-Infer effectively hides communication overhead and maximizes GPU utilization. To adapt to disaggregated attention and FFN modules and minimize data transmission overhead (e.g., token dispatch), MegaScale-Infer provides a high-performance M2N communication library that eliminates unnecessary GPU-to-CPU data copies, group initialization overhead, and GPU synchronization. Experimental results indicate that MegaScale-Infer achieves up to 1.90x higher per-GPU throughput than state-of-the-art solutions. |
| title | MegaScale-Infer: Serving Mixture-of-Experts at Scale with Disaggregated Expert Parallelism |
| topic | Distributed, Parallel, and Cluster Computing Machine Learning |
| url | https://arxiv.org/abs/2504.02263 |