ReXMoE: Reusing Experts with Minimal Overhead in Mixture-of-Experts
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866914104184143872 |
|---|---|
| author | Tan, Zheyue Li, Zhiyuan Yuan, Tao Zhou, Dong Liu, Weilin Zhuang, Yueqing Li, Yadong Niu, Guowei Qin, Cheng Yao, Zhuyu Liu, Congyi Xu, Haiyang Li, Boxun Dai, Guohao Zhao, Bo Wang, Yu |
| author_facet | Tan, Zheyue Li, Zhiyuan Yuan, Tao Zhou, Dong Liu, Weilin Zhuang, Yueqing Li, Yadong Niu, Guowei Qin, Cheng Yao, Zhuyu Liu, Congyi Xu, Haiyang Li, Boxun Dai, Guohao Zhao, Bo Wang, Yu |
| contents | Mixture-of-Experts (MoE) architectures have emerged as a promising approach to scale Large Language Models (LLMs). MoE boosts the efficiency by activating a subset of experts per token. Recent works show that fine-grained experts substantially enriches the combinatorial flexibility of active experts and enhances model expressiveness. However, such a design is fundamentally limited by the layer-local routing mechanism: each layer is restricted to its own expert pool. This requires a careful trade-off between expert dimensionality and routing diversity given fixed parameter budgets. We describe ReXMoE, a novel MoE architecture that improves routing beyond the existing layer-local approaches by allowing routers to reuse experts across adjacent layers. ReXMoE decouples expert dimensionality from per-layer budgets, enabling richer expert combinations without sacrificing individual expert capacity or inflating overall parameters. To this end, we propose a new progressive scaling routing (PSR) strategy to gradually increase the candidate expert pool during training. As a result, ReXMoE improves both language modeling and downstream task performance. Extensive experiments on models ranging from 0.5B to 7B parameters across different architectures demonstrate that ReXMoE consistently improves performance under fixed architectural dimensions, confirming ReXMoE as new design paradigm for parameter-efficient and scalable MoE-based LLMs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_17483 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | ReXMoE: Reusing Experts with Minimal Overhead in Mixture-of-Experts Tan, Zheyue Li, Zhiyuan Yuan, Tao Zhou, Dong Liu, Weilin Zhuang, Yueqing Li, Yadong Niu, Guowei Qin, Cheng Yao, Zhuyu Liu, Congyi Xu, Haiyang Li, Boxun Dai, Guohao Zhao, Bo Wang, Yu Computation and Language Mixture-of-Experts (MoE) architectures have emerged as a promising approach to scale Large Language Models (LLMs). MoE boosts the efficiency by activating a subset of experts per token. Recent works show that fine-grained experts substantially enriches the combinatorial flexibility of active experts and enhances model expressiveness. However, such a design is fundamentally limited by the layer-local routing mechanism: each layer is restricted to its own expert pool. This requires a careful trade-off between expert dimensionality and routing diversity given fixed parameter budgets. We describe ReXMoE, a novel MoE architecture that improves routing beyond the existing layer-local approaches by allowing routers to reuse experts across adjacent layers. ReXMoE decouples expert dimensionality from per-layer budgets, enabling richer expert combinations without sacrificing individual expert capacity or inflating overall parameters. To this end, we propose a new progressive scaling routing (PSR) strategy to gradually increase the candidate expert pool during training. As a result, ReXMoE improves both language modeling and downstream task performance. Extensive experiments on models ranging from 0.5B to 7B parameters across different architectures demonstrate that ReXMoE consistently improves performance under fixed architectural dimensions, confirming ReXMoE as new design paradigm for parameter-efficient and scalable MoE-based LLMs. |
| title | ReXMoE: Reusing Experts with Minimal Overhead in Mixture-of-Experts |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2510.17483 |