Advancing Expert Specialization for Better MoE
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908787342835712 |
|---|---|
| author | Guo, Hongcan Lu, Haolang Nan, Guoshun Chu, Bolun Zhuang, Jialin Yang, Yuan Che, Wenhao Cao, Xinye Leng, Sicong Cui, Qimei Jiang, Xudong |
| author_facet | Guo, Hongcan Lu, Haolang Nan, Guoshun Chu, Bolun Zhuang, Jialin Yang, Yuan Che, Wenhao Cao, Xinye Leng, Sicong Cui, Qimei Jiang, Xudong |
| contents | Mixture-of-Experts (MoE) models enable efficient scaling of large language models (LLMs) by activating only a subset of experts per input. However, we observe that the commonly used auxiliary load balancing loss often leads to expert overlap and overly uniform routing, which hinders expert specialization and degrades overall performance during post-training. To address this, we propose a simple yet effective solution that introduces two complementary objectives: (1) an orthogonality loss to encourage experts to process distinct types of tokens, and (2) a variance loss to encourage more discriminative routing decisions. Gradient-level analysis demonstrates that these objectives are compatible with the existing auxiliary loss and contribute to optimizing the training process. Experimental results over various model architectures and across multiple benchmarks show that our method significantly enhances expert specialization. Notably, our method improves classic MoE baselines with auxiliary loss by up to 23.79%, while also maintaining load balancing in downstream tasks, without any architectural modifications or additional components. We will release our code to contribute to the community. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_22323 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Advancing Expert Specialization for Better MoE Guo, Hongcan Lu, Haolang Nan, Guoshun Chu, Bolun Zhuang, Jialin Yang, Yuan Che, Wenhao Cao, Xinye Leng, Sicong Cui, Qimei Jiang, Xudong Computation and Language 68T07 I.2.7 Mixture-of-Experts (MoE) models enable efficient scaling of large language models (LLMs) by activating only a subset of experts per input. However, we observe that the commonly used auxiliary load balancing loss often leads to expert overlap and overly uniform routing, which hinders expert specialization and degrades overall performance during post-training. To address this, we propose a simple yet effective solution that introduces two complementary objectives: (1) an orthogonality loss to encourage experts to process distinct types of tokens, and (2) a variance loss to encourage more discriminative routing decisions. Gradient-level analysis demonstrates that these objectives are compatible with the existing auxiliary loss and contribute to optimizing the training process. Experimental results over various model architectures and across multiple benchmarks show that our method significantly enhances expert specialization. Notably, our method improves classic MoE baselines with auxiliary loss by up to 23.79%, while also maintaining load balancing in downstream tasks, without any architectural modifications or additional components. We will release our code to contribute to the community. |
| title | Advancing Expert Specialization for Better MoE |
| topic | Computation and Language 68T07 I.2.7 |
| url | https://arxiv.org/abs/2505.22323 |