Innovator: Scientific Continued Pretraining with Fine-grained MoE Upcycling
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909849110970368 |
|---|---|
| author | Liao, Ning Wang, Xiaoxing Lin, Zehao Guo, Weiyang Hong, Feng Song, Shixiang Yu, Geng Zhao, Zihua Xie, Sitao Wei, Longxuan Jin, Xiangqi Qin, Xiaohan Ma, Jiale Chen, Kai Yao, Jiangchao Lin, Zhouhan Yan, Junchi Li, Zhiyu Xiong, Feiyu Wang, Yanfeng Zhang, Linfeng |
| author_facet | Liao, Ning Wang, Xiaoxing Lin, Zehao Guo, Weiyang Hong, Feng Song, Shixiang Yu, Geng Zhao, Zihua Xie, Sitao Wei, Longxuan Jin, Xiangqi Qin, Xiaohan Ma, Jiale Chen, Kai Yao, Jiangchao Lin, Zhouhan Yan, Junchi Li, Zhiyu Xiong, Feiyu Wang, Yanfeng Zhang, Linfeng |
| contents | A large language model (LLM) with knowledge in both scientific and general tasks is the foundation of science general intelligence. However, directly continued pretraining an LLM using science data usually leads to catastrophic forgetting, which indicates severe degradation in general ability. In this report, we present Innovator, which solves this problem by upcycling a pre-trained dense LLM into a fine-grained Mixtures-of-Experts model during continued pretraining, where different experts are expected to learn science knowledge in different disciplines, and a shared expert is utilized for general tasks. Innovator introduces a four-stage upcycle training paradigm: (1) Scientific Expert Induction on discipline-specific data, (2) Fine-grained Expert Splitting via FFN dimension decomposition, (3) Science-Aware Routing warmup, and (4) Generalist-Scientist Integration training on hybrid datasets. Such a paradigm enables knowledge in the general domain, and different scientific disciplines can be decoupled, avoiding the negative influence among knowledge in different domains. With 53.3B total parameters and 13.3B activated, Innovator extends Qwen2.5-7B using a shared general expert and 64 specialized scientific experts with 8 activated. Trained on 300B tokens with tri-level quality-controlled data, Innovator achieves 25% average improvement across 30 scientific tasks with a win rate as 70%, while retaining 99% performance in general tasks. Furthermore, Innovator-Reason, which is post-trained from Innovator for reasoning boosting, exhibits excellent reasoning performance in solving complex scientific problems with improvements over 30%. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_18671 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Innovator: Scientific Continued Pretraining with Fine-grained MoE Upcycling Liao, Ning Wang, Xiaoxing Lin, Zehao Guo, Weiyang Hong, Feng Song, Shixiang Yu, Geng Zhao, Zihua Xie, Sitao Wei, Longxuan Jin, Xiangqi Qin, Xiaohan Ma, Jiale Chen, Kai Yao, Jiangchao Lin, Zhouhan Yan, Junchi Li, Zhiyu Xiong, Feiyu Wang, Yanfeng Zhang, Linfeng Machine Learning Artificial Intelligence A large language model (LLM) with knowledge in both scientific and general tasks is the foundation of science general intelligence. However, directly continued pretraining an LLM using science data usually leads to catastrophic forgetting, which indicates severe degradation in general ability. In this report, we present Innovator, which solves this problem by upcycling a pre-trained dense LLM into a fine-grained Mixtures-of-Experts model during continued pretraining, where different experts are expected to learn science knowledge in different disciplines, and a shared expert is utilized for general tasks. Innovator introduces a four-stage upcycle training paradigm: (1) Scientific Expert Induction on discipline-specific data, (2) Fine-grained Expert Splitting via FFN dimension decomposition, (3) Science-Aware Routing warmup, and (4) Generalist-Scientist Integration training on hybrid datasets. Such a paradigm enables knowledge in the general domain, and different scientific disciplines can be decoupled, avoiding the negative influence among knowledge in different domains. With 53.3B total parameters and 13.3B activated, Innovator extends Qwen2.5-7B using a shared general expert and 64 specialized scientific experts with 8 activated. Trained on 300B tokens with tri-level quality-controlled data, Innovator achieves 25% average improvement across 30 scientific tasks with a win rate as 70%, while retaining 99% performance in general tasks. Furthermore, Innovator-Reason, which is post-trained from Innovator for reasoning boosting, exhibits excellent reasoning performance in solving complex scientific problems with improvements over 30%. |
| title | Innovator: Scientific Continued Pretraining with Fine-grained MoE Upcycling |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2507.18671 |