CompeteSMoE -- Effective Training of Sparse Mixture of Experts via Competition
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866914666409623552 |
|---|---|
| author | Pham, Quang Do, Giang Nguyen, Huy Nguyen, TrungTin Liu, Chenghao Sartipi, Mina Nguyen, Binh T. Ramasamy, Savitha Li, Xiaoli Hoi, Steven Ho, Nhat |
| author_facet | Pham, Quang Do, Giang Nguyen, Huy Nguyen, TrungTin Liu, Chenghao Sartipi, Mina Nguyen, Binh T. Ramasamy, Savitha Li, Xiaoli Hoi, Steven Ho, Nhat |
| contents | Sparse mixture of experts (SMoE) offers an appealing solution to scale up the model complexity beyond the mean of increasing the network's depth or width. However, effective training of SMoE has proven to be challenging due to the representation collapse issue, which causes parameter redundancy and limited representation potentials. In this work, we propose a competition mechanism to address this fundamental challenge of representation collapse. By routing inputs only to experts with the highest neural response, we show that, under mild assumptions, competition enjoys the same convergence rate as the optimal estimator. We further propose CompeteSMoE, an effective and efficient algorithm to train large language models by deploying a simple router that predicts the competition outcomes. Consequently, CompeteSMoE enjoys strong performance gains from the competition routing policy while having low computation overheads. Our extensive empirical evaluations on two transformer architectures and a wide range of tasks demonstrate the efficacy, robustness, and scalability of CompeteSMoE compared to state-of-the-art SMoE strategies. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_02526 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | CompeteSMoE -- Effective Training of Sparse Mixture of Experts via Competition Pham, Quang Do, Giang Nguyen, Huy Nguyen, TrungTin Liu, Chenghao Sartipi, Mina Nguyen, Binh T. Ramasamy, Savitha Li, Xiaoli Hoi, Steven Ho, Nhat Machine Learning Sparse mixture of experts (SMoE) offers an appealing solution to scale up the model complexity beyond the mean of increasing the network's depth or width. However, effective training of SMoE has proven to be challenging due to the representation collapse issue, which causes parameter redundancy and limited representation potentials. In this work, we propose a competition mechanism to address this fundamental challenge of representation collapse. By routing inputs only to experts with the highest neural response, we show that, under mild assumptions, competition enjoys the same convergence rate as the optimal estimator. We further propose CompeteSMoE, an effective and efficient algorithm to train large language models by deploying a simple router that predicts the competition outcomes. Consequently, CompeteSMoE enjoys strong performance gains from the competition routing policy while having low computation overheads. Our extensive empirical evaluations on two transformer architectures and a wide range of tasks demonstrate the efficacy, robustness, and scalability of CompeteSMoE compared to state-of-the-art SMoE strategies. |
| title | CompeteSMoE -- Effective Training of Sparse Mixture of Experts via Competition |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2402.02526 |