Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2601.14327 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914369274642432 |
|---|---|
| author | ai, YuanLab. : Wu, Shawn Luo, Jiangang Chen, Darcy Wang, Sean Li, Louie Wang, Allen Zhao, Xudong Yu, Tong Li, Bach Shen, Joseph Ma, Gawain Jia, Jasper Mao, Marcus Wang, Claire He, Hunter Wang, Carol Zhang, Zera Wang, Jason Shen, Chonly Zhang, Leo Chen, Logan Meng, Qasim Gong, James Zhao, Daniel Zheng, Penn Zhu, Owen |
| author_facet | ai, YuanLab. : Wu, Shawn Luo, Jiangang Chen, Darcy Wang, Sean Li, Louie Wang, Allen Zhao, Xudong Yu, Tong Li, Bach Shen, Joseph Ma, Gawain Jia, Jasper Mao, Marcus Wang, Claire He, Hunter Wang, Carol Zhang, Zera Wang, Jason Shen, Chonly Zhang, Leo Chen, Logan Meng, Qasim Gong, James Zhao, Daniel Zheng, Penn Zhu, Owen |
| contents | We introduce Yuan3.0 Ultra, an open-source Mixture-of-Experts (MoE) large language model featuring 68.8B activated parameters and 1010B total parameters, specially designed to enhance performance on enterprise scenarios tasks while maintaining competitive capabilities on general purpose tasks. We propose Layer-Adaptive Expert Pruning (LAEP) algorithm designed for the pre-training stage of MoE LLMs. In contrast to previous expert pruning approaches that operate primarily in the post-training phase, the proposed algorithm enhances training efficiency by selectively pruning underutilized experts and reorganizing experts across computing devices according to token distribution statistics. Comprehensive experiments demonstrate that LAEP effectively reduces model size and substantially improves pre-training efficiency. When pre-training Yuan3.0 Ultra from scratch original with 1515B parameters, this algorithm delivers a 49\% boost in pre-training efficiency and a 33.3\% reduction in total parameters, while preserving the model's outstanding multi-domain performance. On enterprise scenario benchmarks including Docmatix, ChatRAG, SummEval and MMTab, Yuan3.0 Ultra achieves leading accuracy. The model and codes are publicly available at https://github.com/Yuan-lab-LLM/Yuan3.0-Ultra. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_14327 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Yuan3.0 Ultra: A Trillion-Parameter Enterprise-Oriented MoE LLM ai, YuanLab. : Wu, Shawn Luo, Jiangang Chen, Darcy Wang, Sean Li, Louie Wang, Allen Zhao, Xudong Yu, Tong Li, Bach Shen, Joseph Ma, Gawain Jia, Jasper Mao, Marcus Wang, Claire He, Hunter Wang, Carol Zhang, Zera Wang, Jason Shen, Chonly Zhang, Leo Chen, Logan Meng, Qasim Gong, James Zhao, Daniel Zheng, Penn Zhu, Owen Machine Learning Artificial Intelligence Computation and Language We introduce Yuan3.0 Ultra, an open-source Mixture-of-Experts (MoE) large language model featuring 68.8B activated parameters and 1010B total parameters, specially designed to enhance performance on enterprise scenarios tasks while maintaining competitive capabilities on general purpose tasks. We propose Layer-Adaptive Expert Pruning (LAEP) algorithm designed for the pre-training stage of MoE LLMs. In contrast to previous expert pruning approaches that operate primarily in the post-training phase, the proposed algorithm enhances training efficiency by selectively pruning underutilized experts and reorganizing experts across computing devices according to token distribution statistics. Comprehensive experiments demonstrate that LAEP effectively reduces model size and substantially improves pre-training efficiency. When pre-training Yuan3.0 Ultra from scratch original with 1515B parameters, this algorithm delivers a 49\% boost in pre-training efficiency and a 33.3\% reduction in total parameters, while preserving the model's outstanding multi-domain performance. On enterprise scenario benchmarks including Docmatix, ChatRAG, SummEval and MMTab, Yuan3.0 Ultra achieves leading accuracy. The model and codes are publicly available at https://github.com/Yuan-lab-LLM/Yuan3.0-Ultra. |
| title | Yuan3.0 Ultra: A Trillion-Parameter Enterprise-Oriented MoE LLM |
| topic | Machine Learning Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2601.14327 |