Diversifying the Expert Knowledge for Task-Agnostic Pruning in Sparse Mixture-of-Experts
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866910993504796672 |
|---|---|
| author | Zhang, Zeliang Liu, Xiaodong Cheng, Hao Xu, Chenliang Gao, Jianfeng |
| author_facet | Zhang, Zeliang Liu, Xiaodong Cheng, Hao Xu, Chenliang Gao, Jianfeng |
| contents | By increasing model parameters but activating them sparsely when performing a task, the use of Mixture-of-Experts (MoE) architecture significantly improves the performance of Large Language Models (LLMs) without increasing the inference cost. However, the memory consumption due to the growing number of experts presents a challenge to the deployment of these models in many real world settings. Our empirical study reveals that some experts encode redundant knowledge during pre-training. We thus propose a method of grouping and pruning similar experts to improve the model's parameter efficiency. We validate the effectiveness of our method by pruning three state-of-the-art MoE architectures, including Mixtral, Deepseek-MoE, and Qwen. The evaluation shows that our method outperforms other model pruning methods on a range of natural language tasks. We will release our code to facilitate future research. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_09590 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Diversifying the Expert Knowledge for Task-Agnostic Pruning in Sparse Mixture-of-Experts Zhang, Zeliang Liu, Xiaodong Cheng, Hao Xu, Chenliang Gao, Jianfeng Computation and Language Machine Learning By increasing model parameters but activating them sparsely when performing a task, the use of Mixture-of-Experts (MoE) architecture significantly improves the performance of Large Language Models (LLMs) without increasing the inference cost. However, the memory consumption due to the growing number of experts presents a challenge to the deployment of these models in many real world settings. Our empirical study reveals that some experts encode redundant knowledge during pre-training. We thus propose a method of grouping and pruning similar experts to improve the model's parameter efficiency. We validate the effectiveness of our method by pruning three state-of-the-art MoE architectures, including Mixtral, Deepseek-MoE, and Qwen. The evaluation shows that our method outperforms other model pruning methods on a range of natural language tasks. We will release our code to facilitate future research. |
| title | Diversifying the Expert Knowledge for Task-Agnostic Pruning in Sparse Mixture-of-Experts |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2407.09590 |