Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2308.15030 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916264188837888 |
|---|---|
| author | Kong, Rui Li, Yuanchun Feng, Qingtian Wang, Weijun Ye, Xiaozhou Ouyang, Ye Kong, Linghe Liu, Yunxin |
| author_facet | Kong, Rui Li, Yuanchun Feng, Qingtian Wang, Weijun Ye, Xiaozhou Ouyang, Ye Kong, Linghe Liu, Yunxin |
| contents | Mixture of experts (MoE) is a popular technique to improve capacity of Large Language Models (LLMs) with conditionally-activated parallel experts. However, serving MoE models on memory-constrained devices is challenging due to the large parameter size. Typical solutions such as memory swapping or expert pruning may lead to significantly higher latency or severe accuracy loss. In this paper, we introduce SwapMoE, a framework for efficient serving of MoE-based large language models with tunable memory budgets. The main idea of SwapMoE is to keep a small dynamic set of important experts, namely Virtual Experts, in the main memory for inference, while seamlessly maintaining how the Virtual Experts map to the actual experts. Experiments have shown that SwapMoE can reduce the memory footprint while maintaining reasonable accuracy. For example, on text summarization tasks with Switch Transformer, SwapMoE can reduce the memory consumption from 14.2 GiB to 4.7 GiB, together with 50\% latency reduction and a slight Rouge-2 score drop of 0.041. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2308_15030 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | SwapMoE: Serving Off-the-shelf MoE-based Large Language Models with Tunable Memory Budget Kong, Rui Li, Yuanchun Feng, Qingtian Wang, Weijun Ye, Xiaozhou Ouyang, Ye Kong, Linghe Liu, Yunxin Artificial Intelligence Mixture of experts (MoE) is a popular technique to improve capacity of Large Language Models (LLMs) with conditionally-activated parallel experts. However, serving MoE models on memory-constrained devices is challenging due to the large parameter size. Typical solutions such as memory swapping or expert pruning may lead to significantly higher latency or severe accuracy loss. In this paper, we introduce SwapMoE, a framework for efficient serving of MoE-based large language models with tunable memory budgets. The main idea of SwapMoE is to keep a small dynamic set of important experts, namely Virtual Experts, in the main memory for inference, while seamlessly maintaining how the Virtual Experts map to the actual experts. Experiments have shown that SwapMoE can reduce the memory footprint while maintaining reasonable accuracy. For example, on text summarization tasks with Switch Transformer, SwapMoE can reduce the memory consumption from 14.2 GiB to 4.7 GiB, together with 50\% latency reduction and a slight Rouge-2 score drop of 0.041. |
| title | SwapMoE: Serving Off-the-shelf MoE-based Large Language Models with Tunable Memory Budget |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2308.15030 |