Expert Divergence Learning for MoE-based Language Models
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917300325580800 |
|---|---|
| author | Li, Jiaang Chen, Haibin Liu, Langming Yuan, Yujin Wang, Yadao Zhang, Yizhen Yu, Chengting Tong, Xin Zhang, Weidong Liu, Shilei Su, Wenbo Zheng, Bo |
| author_facet | Li, Jiaang Chen, Haibin Liu, Langming Yuan, Yujin Wang, Yadao Zhang, Yizhen Yu, Chengting Tong, Xin Zhang, Weidong Liu, Shilei Su, Wenbo Zheng, Bo |
| contents | The Mixture-of-Experts (MoE) architecture is a powerful technique for scaling language models, yet it often suffers from expert homogenization, where experts learn redundant functionalities, thereby limiting MoE's full potential. To address this, we introduce Expert Divergence Learning, a novel pre-training strategy that explicitly encourages functional specialization among experts. Our method incorporates a label-driven auxiliary loss that leverages domain labels inherent in pre-training corpora to maximize the Jensen-Shannon Divergence between the expert routing distributions of different data domains. This optimization objective guides the model to develop diverged routing policies for varied domains and closer routing policies for the same domain, which leads to emergent and organized expert specialization. We validate our approach by pre-training MoE models of up to 15 billion parameters from scratch. Experimental results demonstrate that models trained with Expert Divergence Learning not only achieve a lower language modeling loss but also exhibit significant performance improvements across a diverse range of downstream benchmarks. Further analysis confirms that our method effectively mitigates expert homogenization and brings greater functional specialization, all with negligible computational overhead during training. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_00054 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Expert Divergence Learning for MoE-based Language Models Li, Jiaang Chen, Haibin Liu, Langming Yuan, Yujin Wang, Yadao Zhang, Yizhen Yu, Chengting Tong, Xin Zhang, Weidong Liu, Shilei Su, Wenbo Zheng, Bo Machine Learning Artificial Intelligence The Mixture-of-Experts (MoE) architecture is a powerful technique for scaling language models, yet it often suffers from expert homogenization, where experts learn redundant functionalities, thereby limiting MoE's full potential. To address this, we introduce Expert Divergence Learning, a novel pre-training strategy that explicitly encourages functional specialization among experts. Our method incorporates a label-driven auxiliary loss that leverages domain labels inherent in pre-training corpora to maximize the Jensen-Shannon Divergence between the expert routing distributions of different data domains. This optimization objective guides the model to develop diverged routing policies for varied domains and closer routing policies for the same domain, which leads to emergent and organized expert specialization. We validate our approach by pre-training MoE models of up to 15 billion parameters from scratch. Experimental results demonstrate that models trained with Expert Divergence Learning not only achieve a lower language modeling loss but also exhibit significant performance improvements across a diverse range of downstream benchmarks. Further analysis confirms that our method effectively mitigates expert homogenization and brings greater functional specialization, all with negligible computational overhead during training. |
| title | Expert Divergence Learning for MoE-based Language Models |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2603.00054 |