Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | https://arxiv.org/abs/2408.10681 |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866917754022395904 |
|---|---|
| author | Wang, An Sun, Xingwu Xie, Ruobing Li, Shuaipeng Zhu, Jiaqi Yang, Zhen Zhao, Pinxue Han, J. N. Kang, Zhanhui Wang, Di Okazaki, Naoaki Xu, Cheng-zhong |
| author_facet | Wang, An Sun, Xingwu Xie, Ruobing Li, Shuaipeng Zhu, Jiaqi Yang, Zhen Zhao, Pinxue Han, J. N. Kang, Zhanhui Wang, Di Okazaki, Naoaki Xu, Cheng-zhong |
| contents | Mixture of Experts (MoE) offers remarkable performance and computational efficiency by selectively activating subsets of model parameters. Traditionally, MoE models use homogeneous experts, each with identical capacity. However, varying complexity in input data necessitates experts with diverse capabilities, while homogeneous MoE hinders effective expert specialization and efficient parameter utilization. In this study, we propose a novel Heterogeneous Mixture of Experts (HMoE), where experts differ in size and thus possess diverse capacities. This heterogeneity allows for more specialized experts to handle varying token complexities more effectively. To address the imbalance in expert activation, we propose a novel training objective that encourages the frequent activation of smaller experts, enhancing computational efficiency and parameter utilization. Extensive experiments demonstrate that HMoE achieves lower loss with fewer activated parameters and outperforms conventional homogeneous MoE models on various pre-training evaluation benchmarks. Codes will be released upon acceptance. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_10681 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | HMoE: Heterogeneous Mixture of Experts for Language Modeling Wang, An Sun, Xingwu Xie, Ruobing Li, Shuaipeng Zhu, Jiaqi Yang, Zhen Zhao, Pinxue Han, J. N. Kang, Zhanhui Wang, Di Okazaki, Naoaki Xu, Cheng-zhong Computation and Language Machine Learning Mixture of Experts (MoE) offers remarkable performance and computational efficiency by selectively activating subsets of model parameters. Traditionally, MoE models use homogeneous experts, each with identical capacity. However, varying complexity in input data necessitates experts with diverse capabilities, while homogeneous MoE hinders effective expert specialization and efficient parameter utilization. In this study, we propose a novel Heterogeneous Mixture of Experts (HMoE), where experts differ in size and thus possess diverse capacities. This heterogeneity allows for more specialized experts to handle varying token complexities more effectively. To address the imbalance in expert activation, we propose a novel training objective that encourages the frequent activation of smaller experts, enhancing computational efficiency and parameter utilization. Extensive experiments demonstrate that HMoE achieves lower loss with fewer activated parameters and outperforms conventional homogeneous MoE models on various pre-training evaluation benchmarks. Codes will be released upon acceptance. |
| title | HMoE: Heterogeneous Mixture of Experts for Language Modeling |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2408.10681 |