EPS-MoE: Expert Pipeline Scheduler for Cost-Efficient MoE Inference
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916549317623808 |
|---|---|
| author | Qian, Yulei Li, Fengcun Ji, Xiangyang Zhao, Xiaoyu Tan, Jianchao Zhang, Kefeng Cai, Xunliang |
| author_facet | Qian, Yulei Li, Fengcun Ji, Xiangyang Zhao, Xiaoyu Tan, Jianchao Zhang, Kefeng Cai, Xunliang |
| contents | The Mixture-of-Experts (MoE) model has emerged as a prominent architecture in the field of Large Language Models (LLMs), providing a better balance between model performance and computational efficiency. However the General Matrix Multiply (GEMM) operations and large parameters introduce challenges related to computational efficiency and communication overhead, which become throughput bottlenecks during inference. Applying a single parallelism strategy like EP, DP, TP or a straightforward combination of them to MoE usually achieves sub-optimal inference throughput. This paper introduces EPS-MoE, a novel expert pipeline scheduler for MoE that surpasses the existing parallelism schemes. Our approach optimizes the computation of MoE FeedForward Network (FFN) modules by dynamically selecting the best kernel implementation of GroupGemm and DenseGemm for different loads and adaptively overlapping these computations with communication, leading to a substantial increase in throughput. Our experimental results demonstrate at most 52.4\% improvement in prefill throughput compared to existing parallel inference methods. Specifically, our method accelerated the highly optimized DeepSeekV2 model from a claimed 100K tokens per second to at least 120K tokens per second. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_12247 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | EPS-MoE: Expert Pipeline Scheduler for Cost-Efficient MoE Inference Qian, Yulei Li, Fengcun Ji, Xiangyang Zhao, Xiaoyu Tan, Jianchao Zhang, Kefeng Cai, Xunliang Computation and Language Distributed, Parallel, and Cluster Computing The Mixture-of-Experts (MoE) model has emerged as a prominent architecture in the field of Large Language Models (LLMs), providing a better balance between model performance and computational efficiency. However the General Matrix Multiply (GEMM) operations and large parameters introduce challenges related to computational efficiency and communication overhead, which become throughput bottlenecks during inference. Applying a single parallelism strategy like EP, DP, TP or a straightforward combination of them to MoE usually achieves sub-optimal inference throughput. This paper introduces EPS-MoE, a novel expert pipeline scheduler for MoE that surpasses the existing parallelism schemes. Our approach optimizes the computation of MoE FeedForward Network (FFN) modules by dynamically selecting the best kernel implementation of GroupGemm and DenseGemm for different loads and adaptively overlapping these computations with communication, leading to a substantial increase in throughput. Our experimental results demonstrate at most 52.4\% improvement in prefill throughput compared to existing parallel inference methods. Specifically, our method accelerated the highly optimized DeepSeekV2 model from a claimed 100K tokens per second to at least 120K tokens per second. |
| title | EPS-MoE: Expert Pipeline Scheduler for Cost-Efficient MoE Inference |
| topic | Computation and Language Distributed, Parallel, and Cluster Computing |
| url | https://arxiv.org/abs/2410.12247 |