Mixture of Universal Experts: Scaling Virtual Width via Depth-Width Transformation
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2026
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866911488306839552 |
|---|---|
| author | Chen, Yilong Gu, Naibin Shang, Junyuan Zhang, Zhenyu Feng, Yuchen Sheng, Jiawei Liu, Tingwen Wang, Shuohuan Sun, Yu Wu, Hua Wang, Haifeng |
| author_facet | Chen, Yilong Gu, Naibin Shang, Junyuan Zhang, Zhenyu Feng, Yuchen Sheng, Jiawei Liu, Tingwen Wang, Shuohuan Sun, Yu Wu, Hua Wang, Haifeng |
| contents | Mixture-of-Experts (MoE) decouples model capacity from per-token computation, yet their scalability remains limited by the physical dimensions of depth and width. To overcome this, we propose Mixture of Universal Experts (MOUE),a MoE generalization introducing a novel scaling dimension: Virtual Width. In general, MoUE aims to reuse a universal layer-agnostic expert pool across layers, converting depth into virtual width under a fixed per-token activation budget. However, two challenges remain: a routing path explosion from recursive expert reuse, and a mismatch between the exposure induced by reuse and the conventional load-balancing objectives. We address these with three core components: a Staggered Rotational Topology for structured expert sharing, a Universal Expert Load Balance for depth-aware exposure correction, and a Universal Router with lightweight trajectory state for coherent multi-step routing. Empirically, MoUE consistently outperforms matched MoE baselines by up to 1.3% across scaling regimes, enables progressive conversion of existing MoE checkpoints with up to 4.2% gains, and reveals a new scaling dimension for MoE architectures. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_04971 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Mixture of Universal Experts: Scaling Virtual Width via Depth-Width Transformation Chen, Yilong Gu, Naibin Shang, Junyuan Zhang, Zhenyu Feng, Yuchen Sheng, Jiawei Liu, Tingwen Wang, Shuohuan Sun, Yu Wu, Hua Wang, Haifeng Machine Learning Artificial Intelligence Computation and Language Mixture-of-Experts (MoE) decouples model capacity from per-token computation, yet their scalability remains limited by the physical dimensions of depth and width. To overcome this, we propose Mixture of Universal Experts (MOUE),a MoE generalization introducing a novel scaling dimension: Virtual Width. In general, MoUE aims to reuse a universal layer-agnostic expert pool across layers, converting depth into virtual width under a fixed per-token activation budget. However, two challenges remain: a routing path explosion from recursive expert reuse, and a mismatch between the exposure induced by reuse and the conventional load-balancing objectives. We address these with three core components: a Staggered Rotational Topology for structured expert sharing, a Universal Expert Load Balance for depth-aware exposure correction, and a Universal Router with lightweight trajectory state for coherent multi-step routing. Empirically, MoUE consistently outperforms matched MoE baselines by up to 1.3% across scaling regimes, enables progressive conversion of existing MoE checkpoints with up to 4.2% gains, and reveals a new scaling dimension for MoE architectures. |
| title | Mixture of Universal Experts: Scaling Virtual Width via Depth-Width Transformation |
| topic | Machine Learning Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2603.04971 |