DSMoE: Matrix-Partitioned Experts with Dynamic Routing for Computation-Efficient Dense LLMs
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866908539102953472 |
|---|---|
| author | Lv, Minxuan Su, Zhenpeng Pan, Leiyu Xiong, Yizhe Lin, Zijia Chen, Hui Zhou, Wei Han, Jungong Ding, Guiguang Luo, Cheng Zhang, Di Gai, Kun Hu, Songlin |
| author_facet | Lv, Minxuan Su, Zhenpeng Pan, Leiyu Xiong, Yizhe Lin, Zijia Chen, Hui Zhou, Wei Han, Jungong Ding, Guiguang Luo, Cheng Zhang, Di Gai, Kun Hu, Songlin |
| contents | As large language models continue to scale, computational costs and resource consumption have emerged as significant challenges. While existing sparsification methods like pruning reduce computational overhead, they risk losing model knowledge through parameter removal. This paper proposes DSMoE (Dynamic Sparse Mixture-of-Experts), a novel approach that achieves sparsification by partitioning pre-trained FFN layers into computational blocks. We implement adaptive expert routing using sigmoid activation and straight-through estimators, enabling tokens to flexibly access different aspects of model knowledge based on input complexity. Additionally, we introduce a sparsity loss term to balance performance and computational efficiency. Extensive experiments on LLaMA models demonstrate that under equivalent computational constraints, DSMoE achieves superior performance compared to existing pruning and MoE approaches across language modeling and downstream tasks, particularly excelling in generation tasks. Analysis reveals that DSMoE learns distinctive layerwise activation patterns, providing new insights for future MoE architecture design. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_12455 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | DSMoE: Matrix-Partitioned Experts with Dynamic Routing for Computation-Efficient Dense LLMs Lv, Minxuan Su, Zhenpeng Pan, Leiyu Xiong, Yizhe Lin, Zijia Chen, Hui Zhou, Wei Han, Jungong Ding, Guiguang Luo, Cheng Zhang, Di Gai, Kun Hu, Songlin Computation and Language As large language models continue to scale, computational costs and resource consumption have emerged as significant challenges. While existing sparsification methods like pruning reduce computational overhead, they risk losing model knowledge through parameter removal. This paper proposes DSMoE (Dynamic Sparse Mixture-of-Experts), a novel approach that achieves sparsification by partitioning pre-trained FFN layers into computational blocks. We implement adaptive expert routing using sigmoid activation and straight-through estimators, enabling tokens to flexibly access different aspects of model knowledge based on input complexity. Additionally, we introduce a sparsity loss term to balance performance and computational efficiency. Extensive experiments on LLaMA models demonstrate that under equivalent computational constraints, DSMoE achieves superior performance compared to existing pruning and MoE approaches across language modeling and downstream tasks, particularly excelling in generation tasks. Analysis reveals that DSMoE learns distinctive layerwise activation patterns, providing new insights for future MoE architecture design. |
| title | DSMoE: Matrix-Partitioned Experts with Dynamic Routing for Computation-Efficient Dense LLMs |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2502.12455 |