DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910294718021632 |
|---|---|
| author | Dai, Damai Deng, Chengqi Zhao, Chenggang Xu, R. X. Gao, Huazuo Chen, Deli Li, Jiashi Zeng, Wangding Yu, Xingkai Wu, Y. Xie, Zhenda Li, Y. K. Huang, Panpan Luo, Fuli Ruan, Chong Sui, Zhifang Liang, Wenfeng |
| author_facet | Dai, Damai Deng, Chengqi Zhao, Chenggang Xu, R. X. Gao, Huazuo Chen, Deli Li, Jiashi Zeng, Wangding Yu, Xingkai Wu, Y. Xie, Zhenda Li, Y. K. Huang, Panpan Luo, Fuli Ruan, Chong Sui, Zhifang Liang, Wenfeng |
| contents | In the era of large language models, Mixture-of-Experts (MoE) is a promising architecture for managing computational costs when scaling up model parameters. However, conventional MoE architectures like GShard, which activate the top-$K$ out of $N$ experts, face challenges in ensuring expert specialization, i.e. each expert acquires non-overlapping and focused knowledge. In response, we propose the DeepSeekMoE architecture towards ultimate expert specialization. It involves two principal strategies: (1) finely segmenting the experts into $mN$ ones and activating $mK$ from them, allowing for a more flexible combination of activated experts; (2) isolating $K_s$ experts as shared ones, aiming at capturing common knowledge and mitigating redundancy in routed experts. Starting from a modest scale with 2B parameters, we demonstrate that DeepSeekMoE 2B achieves comparable performance with GShard 2.9B, which has 1.5 times the expert parameters and computation. In addition, DeepSeekMoE 2B nearly approaches the performance of its dense counterpart with the same number of total parameters, which set the upper bound of MoE models. Subsequently, we scale up DeepSeekMoE to 16B parameters and show that it achieves comparable performance with LLaMA2 7B, with only about 40% of computations. Further, our preliminary efforts to scale up DeepSeekMoE to 145B parameters consistently validate its substantial advantages over the GShard architecture, and show its performance comparable with DeepSeek 67B, using only 28.5% (maybe even 18.2%) of computations. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2401_06066 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models Dai, Damai Deng, Chengqi Zhao, Chenggang Xu, R. X. Gao, Huazuo Chen, Deli Li, Jiashi Zeng, Wangding Yu, Xingkai Wu, Y. Xie, Zhenda Li, Y. K. Huang, Panpan Luo, Fuli Ruan, Chong Sui, Zhifang Liang, Wenfeng Computation and Language In the era of large language models, Mixture-of-Experts (MoE) is a promising architecture for managing computational costs when scaling up model parameters. However, conventional MoE architectures like GShard, which activate the top-$K$ out of $N$ experts, face challenges in ensuring expert specialization, i.e. each expert acquires non-overlapping and focused knowledge. In response, we propose the DeepSeekMoE architecture towards ultimate expert specialization. It involves two principal strategies: (1) finely segmenting the experts into $mN$ ones and activating $mK$ from them, allowing for a more flexible combination of activated experts; (2) isolating $K_s$ experts as shared ones, aiming at capturing common knowledge and mitigating redundancy in routed experts. Starting from a modest scale with 2B parameters, we demonstrate that DeepSeekMoE 2B achieves comparable performance with GShard 2.9B, which has 1.5 times the expert parameters and computation. In addition, DeepSeekMoE 2B nearly approaches the performance of its dense counterpart with the same number of total parameters, which set the upper bound of MoE models. Subsequently, we scale up DeepSeekMoE to 16B parameters and show that it achieves comparable performance with LLaMA2 7B, with only about 40% of computations. Further, our preliminary efforts to scale up DeepSeekMoE to 145B parameters consistently validate its substantial advantages over the GShard architecture, and show its performance comparable with DeepSeek 67B, using only 28.5% (maybe even 18.2%) of computations. |
| title | DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2401.06066 |