Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2605.13247 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916011565907968 |
|---|---|
| author | Jin, Linghao Shi, Chufan Wang, Huijuan Wen, Nuan Liu, Zhengzhong Xing, Eric Ma, Xuezhe |
| author_facet | Jin, Linghao Shi, Chufan Wang, Huijuan Wen, Nuan Liu, Zhengzhong Xing, Eric Ma, Xuezhe |
| contents | Sparse Mixture-of-Experts (MoE) models offer a powerful way to scale model size without increasing compute, as per-token FLOPs depend only on k active experts rather than the total pool of E experts. Yet, this asymmetry creates an MoE efficiency paradox in practice: adding more experts balloons memory and communication costs, making actual training inefficient. We argue that this bottleneck arises in part because current MoE training allocates too many experts from the beginning, even though early-stage data may not fully utilize such capacity. Motivated by this, we propose EMO, a simple progressive training framework that treats MoE capacity as expandable memory and grows the expert pool over the course of training. EMO explicitly models sparsity in scaling law to derive stage-wise compute-optimal token budgets for progressive expansion. Empirical results show that EMO matches the performance of a fixed-expert setup in large-scale experiments while improving wall-clock efficiency. It offers a surprisingly simple yet effective path to scalable MoE training, preserving the benefits of large expert pools while reducing both training time and GPU cost. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_13247 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | EMO: Frustratingly Easy Progressive Training of Extendable MoE Jin, Linghao Shi, Chufan Wang, Huijuan Wen, Nuan Liu, Zhengzhong Xing, Eric Ma, Xuezhe Machine Learning Sparse Mixture-of-Experts (MoE) models offer a powerful way to scale model size without increasing compute, as per-token FLOPs depend only on k active experts rather than the total pool of E experts. Yet, this asymmetry creates an MoE efficiency paradox in practice: adding more experts balloons memory and communication costs, making actual training inefficient. We argue that this bottleneck arises in part because current MoE training allocates too many experts from the beginning, even though early-stage data may not fully utilize such capacity. Motivated by this, we propose EMO, a simple progressive training framework that treats MoE capacity as expandable memory and grows the expert pool over the course of training. EMO explicitly models sparsity in scaling law to derive stage-wise compute-optimal token budgets for progressive expansion. Empirical results show that EMO matches the performance of a fixed-expert setup in large-scale experiments while improving wall-clock efficiency. It offers a surprisingly simple yet effective path to scalable MoE training, preserving the benefits of large expert pools while reducing both training time and GPU cost. |
| title | EMO: Frustratingly Easy Progressive Training of Extendable MoE |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2605.13247 |