Mixture-of-Experts Can Surpass Dense LLMs Under Strictly Equal Resource
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910227021955072 |
|---|---|
| author | Li, Houyi Lo, Ka Man Xuyang, Shijie Wang, Ziqi Zheng, Wenzhen Zhang, Haocheng Li, Zhao Zhou, Shuigeng Zhang, Xiangyu Jiang, Daxin |
| author_facet | Li, Houyi Lo, Ka Man Xuyang, Shijie Wang, Ziqi Zheng, Wenzhen Zhang, Haocheng Li, Zhao Zhou, Shuigeng Zhang, Xiangyu Jiang, Daxin |
| contents | Mixture-of-Experts (MoE) language models dramatically expand model capacity and achieve remarkable performance without increasing per-token compute. However, can MoEs surpass dense architectures under strictly equal resource constraints -- that is, when the total parameter count, training compute, and data budget are identical? This question remains under-explored despite its significant practical value and potential. In this paper, we propose a novel perspective and methodological framework to study this question thoroughly. First, we comprehensively investigate the architecture of MoEs and achieve an optimal model design that maximizes the performance. Based on this, we subsequently find that an MoE model with activation rate in an optimal region is able to outperform its dense counterpart under the same total parameter, training compute and data resource. More importantly, this optimal region remains consistent across different model sizes. Although additional amount of data turns out to be a trade-off for enhanced performance, we show that this can be resolved via reusing data. We validate our findings through extensive experiments, training nearly 200 language models at 2B scale and over 50 at 7B scale, cumulatively processing 50 trillion tokens. All model checkpoints are publicly available. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_12119 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Mixture-of-Experts Can Surpass Dense LLMs Under Strictly Equal Resource Li, Houyi Lo, Ka Man Xuyang, Shijie Wang, Ziqi Zheng, Wenzhen Zhang, Haocheng Li, Zhao Zhou, Shuigeng Zhang, Xiangyu Jiang, Daxin Computation and Language Artificial Intelligence Mixture-of-Experts (MoE) language models dramatically expand model capacity and achieve remarkable performance without increasing per-token compute. However, can MoEs surpass dense architectures under strictly equal resource constraints -- that is, when the total parameter count, training compute, and data budget are identical? This question remains under-explored despite its significant practical value and potential. In this paper, we propose a novel perspective and methodological framework to study this question thoroughly. First, we comprehensively investigate the architecture of MoEs and achieve an optimal model design that maximizes the performance. Based on this, we subsequently find that an MoE model with activation rate in an optimal region is able to outperform its dense counterpart under the same total parameter, training compute and data resource. More importantly, this optimal region remains consistent across different model sizes. Although additional amount of data turns out to be a trade-off for enhanced performance, we show that this can be resolved via reusing data. We validate our findings through extensive experiments, training nearly 200 language models at 2B scale and over 50 at 7B scale, cumulatively processing 50 trillion tokens. All model checkpoints are publicly available. |
| title | Mixture-of-Experts Can Surpass Dense LLMs Under Strictly Equal Resource |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2506.12119 |