Marco-MoE: Open Multilingual Mixture-of-Expert Language Models with Efficient Upcycling
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914514322063360 |
|---|---|
| author | Jiang, Fan Zhao, Yu Lyu, Chenyang Shi, Tianqi Du, Yichao Jiang, Feihu Wang, Longyue Luo, Weihua |
| author_facet | Jiang, Fan Zhao, Yu Lyu, Chenyang Shi, Tianqi Du, Yichao Jiang, Feihu Wang, Longyue Luo, Weihua |
| contents | We present Marco-MoE, a suite of fully open multilingual sparse Mixture-of-Experts (MoE) models. Marco-MoE features a highly sparse design in which only around 5\% of the total parameters are activated per input token. This extreme sparsity, combined with upcycling from dense models, enables efficient pre-training on 5T tokens. Our models surpass similarly-sized competitors on English and multilingual benchmarks, achieving a best-in-class performance-to-compute ratio. We further post-train these models to create Marco-MoE-\textsc{Instruct} variants, which surpass the performance of competing models possessing $3$--$14\times$ more activated parameters. Our analysis reveals that Marco-MoE learns structured expert activation patterns shared across related languages, while maintaining highly specialized utilization for linguistically isolated ones. We further show that Marco-MoE allows for scalable language expansion without the interference typical of dense models. To support the community, we disclose our full training datasets, recipes, and model weights. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_25578 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Marco-MoE: Open Multilingual Mixture-of-Expert Language Models with Efficient Upcycling Jiang, Fan Zhao, Yu Lyu, Chenyang Shi, Tianqi Du, Yichao Jiang, Feihu Wang, Longyue Luo, Weihua Computation and Language Artificial Intelligence We present Marco-MoE, a suite of fully open multilingual sparse Mixture-of-Experts (MoE) models. Marco-MoE features a highly sparse design in which only around 5\% of the total parameters are activated per input token. This extreme sparsity, combined with upcycling from dense models, enables efficient pre-training on 5T tokens. Our models surpass similarly-sized competitors on English and multilingual benchmarks, achieving a best-in-class performance-to-compute ratio. We further post-train these models to create Marco-MoE-\textsc{Instruct} variants, which surpass the performance of competing models possessing $3$--$14\times$ more activated parameters. Our analysis reveals that Marco-MoE learns structured expert activation patterns shared across related languages, while maintaining highly specialized utilization for linguistically isolated ones. We further show that Marco-MoE allows for scalable language expansion without the interference typical of dense models. To support the community, we disclose our full training datasets, recipes, and model weights. |
| title | Marco-MoE: Open Multilingual Mixture-of-Expert Language Models with Efficient Upcycling |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2604.25578 |