MoE-Pruner: Pruning Mixture-of-Experts Large Language Model using the Hints from Its Router
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917804654985216 |
|---|---|
| author | Xie, Yanyue Zhang, Zhi Zhou, Ding Xie, Cong Song, Ziang Liu, Xin Wang, Yanzhi Lin, Xue Xu, An |
| author_facet | Xie, Yanyue Zhang, Zhi Zhou, Ding Xie, Cong Song, Ziang Liu, Xin Wang, Yanzhi Lin, Xue Xu, An |
| contents | Mixture-of-Experts (MoE) architectures face challenges such as high memory consumption and redundancy in experts. Pruning MoE can reduce network weights while maintaining model performance. Motivated by the recent observation of emergent large magnitude features in Large Language Models (LLM) and MoE routing policy, we propose MoE-Pruner, a method that prunes weights with the smallest magnitudes multiplied by the corresponding input activations and router weights, on each output neuron. Our pruning method is one-shot, requiring no retraining or weight updates. We evaluate our method on Mixtral-8x7B and Mixtral-8x22B across multiple language benchmarks. Experimental results show that our pruning method significantly outperforms state-of-the-art LLM pruning methods. Furthermore, our pruned MoE models can benefit from a pretrained teacher model through expert-wise knowledge distillation, improving performance post-pruning. Experimental results demonstrate that the Mixtral-8x7B model with 50% sparsity maintains 99% of the performance of the original model after the expert-wise knowledge distillation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_12013 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | MoE-Pruner: Pruning Mixture-of-Experts Large Language Model using the Hints from Its Router Xie, Yanyue Zhang, Zhi Zhou, Ding Xie, Cong Song, Ziang Liu, Xin Wang, Yanzhi Lin, Xue Xu, An Computation and Language Artificial Intelligence Machine Learning Mixture-of-Experts (MoE) architectures face challenges such as high memory consumption and redundancy in experts. Pruning MoE can reduce network weights while maintaining model performance. Motivated by the recent observation of emergent large magnitude features in Large Language Models (LLM) and MoE routing policy, we propose MoE-Pruner, a method that prunes weights with the smallest magnitudes multiplied by the corresponding input activations and router weights, on each output neuron. Our pruning method is one-shot, requiring no retraining or weight updates. We evaluate our method on Mixtral-8x7B and Mixtral-8x22B across multiple language benchmarks. Experimental results show that our pruning method significantly outperforms state-of-the-art LLM pruning methods. Furthermore, our pruned MoE models can benefit from a pretrained teacher model through expert-wise knowledge distillation, improving performance post-pruning. Experimental results demonstrate that the Mixtral-8x7B model with 50% sparsity maintains 99% of the performance of the original model after the expert-wise knowledge distillation. |
| title | MoE-Pruner: Pruning Mixture-of-Experts Large Language Model using the Hints from Its Router |
| topic | Computation and Language Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2410.12013 |