$\infty$-MoE: Generalizing Mixture of Experts to Infinite Experts
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2026
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866910000067117056 |
|---|---|
| author | Takashiro, Shota Kojima, Takeshi Taniguchi, Shohei Iwasawa, Yusuke Matsuo, Yutaka |
| author_facet | Takashiro, Shota Kojima, Takeshi Taniguchi, Shohei Iwasawa, Yusuke Matsuo, Yutaka |
| contents | The Mixture of Experts (MoE) selects a few feed-forward networks (FFNs) per token, achieving an effective trade-off between computational cost and performance. In conventional MoE, each expert is treated as entirely independent, and experts are combined in a discrete space. As a result, when the number of experts increases, it becomes difficult to train each expert effectively. To stabilize training while increasing the number of experts, we propose $\infty$-MoE that selects a portion of the parameters of large FFNs based on continuous values sampled for each token. By considering experts in a continuous space, this approach allows for an infinite number of experts while maintaining computational efficiency. Experiments show that a GPT-2 Small-based $\infty$-MoE model, with 129M active and 186M total parameters, achieves comparable performance to a dense GPT-2 Medium with 350M parameters. Adjusting the number of sampled experts at inference time allows for a flexible trade-off between accuracy and speed, with an improvement of up to 2.5\% in accuracy over conventional MoE. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_17680 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | $\infty$-MoE: Generalizing Mixture of Experts to Infinite Experts Takashiro, Shota Kojima, Takeshi Taniguchi, Shohei Iwasawa, Yusuke Matsuo, Yutaka Machine Learning Computation and Language The Mixture of Experts (MoE) selects a few feed-forward networks (FFNs) per token, achieving an effective trade-off between computational cost and performance. In conventional MoE, each expert is treated as entirely independent, and experts are combined in a discrete space. As a result, when the number of experts increases, it becomes difficult to train each expert effectively. To stabilize training while increasing the number of experts, we propose $\infty$-MoE that selects a portion of the parameters of large FFNs based on continuous values sampled for each token. By considering experts in a continuous space, this approach allows for an infinite number of experts while maintaining computational efficiency. Experiments show that a GPT-2 Small-based $\infty$-MoE model, with 129M active and 186M total parameters, achieves comparable performance to a dense GPT-2 Medium with 350M parameters. Adjusting the number of sampled experts at inference time allows for a flexible trade-off between accuracy and speed, with an improvement of up to 2.5\% in accuracy over conventional MoE. |
| title | $\infty$-MoE: Generalizing Mixture of Experts to Infinite Experts |
| topic | Machine Learning Computation and Language |
| url | https://arxiv.org/abs/2601.17680 |