XMoE: Sparse Models with Fine-grained and Adaptive Expert Selection
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866911885488553984 |
|---|---|
| author | Yang, Yuanhang Qi, Shiyi Gu, Wenchao Wang, Chaozheng Gao, Cuiyun Xu, Zenglin |
| author_facet | Yang, Yuanhang Qi, Shiyi Gu, Wenchao Wang, Chaozheng Gao, Cuiyun Xu, Zenglin |
| contents | Sparse models, including sparse Mixture-of-Experts (MoE) models, have emerged as an effective approach for scaling Transformer models. However, they often suffer from computational inefficiency since a significant number of parameters are unnecessarily involved in computations via multiplying values by zero or low activation values. To address this issue, we present \tool, a novel MoE designed to enhance both the efficacy and efficiency of sparse MoE models. \tool leverages small experts and a threshold-based router to enable tokens to selectively engage only essential parameters. Our extensive experiments on language modeling and machine translation tasks demonstrate that \tool can enhance model performance while decreasing the computation load at MoE layers by over 50\% without sacrificing performance. Furthermore, we present the versatility of \tool by applying it to dense models, enabling sparse computation during inference. We provide a comprehensive analysis and make our code available at https://github.com/ysngki/XMoE. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_18926 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | XMoE: Sparse Models with Fine-grained and Adaptive Expert Selection Yang, Yuanhang Qi, Shiyi Gu, Wenchao Wang, Chaozheng Gao, Cuiyun Xu, Zenglin Machine Learning Computation and Language Sparse models, including sparse Mixture-of-Experts (MoE) models, have emerged as an effective approach for scaling Transformer models. However, they often suffer from computational inefficiency since a significant number of parameters are unnecessarily involved in computations via multiplying values by zero or low activation values. To address this issue, we present \tool, a novel MoE designed to enhance both the efficacy and efficiency of sparse MoE models. \tool leverages small experts and a threshold-based router to enable tokens to selectively engage only essential parameters. Our extensive experiments on language modeling and machine translation tasks demonstrate that \tool can enhance model performance while decreasing the computation load at MoE layers by over 50\% without sacrificing performance. Furthermore, we present the versatility of \tool by applying it to dense models, enabling sparse computation during inference. We provide a comprehensive analysis and make our code available at https://github.com/ysngki/XMoE. |
| title | XMoE: Sparse Models with Fine-grained and Adaptive Expert Selection |
| topic | Machine Learning Computation and Language |
| url | https://arxiv.org/abs/2403.18926 |