Cover Image

Innovator: Scientific Continued Pretraining with Fine-grained MoE Upcycling

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Liao, Ning, Wang, Xiaoxing, Lin, Zehao, Guo, Weiyang, Hong, Feng, Song, Shixiang, Yu, Geng, Zhao, Zihua, Xie, Sitao, Wei, Longxuan, Jin, Xiangqi, Qin, Xiaohan, Ma, Jiale, Chen, Kai, Yao, Jiangchao, Lin, Zhouhan, Yan, Junchi, Li, Zhiyu, Xiong, Feiyu, Wang, Yanfeng, Zhang, Linfeng
Format:	Preprint
Published:	2025
Subjects:	Machine Learning Artificial Intelligence
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

FineRMoE: Dimension Expansion for Finer-Grained Expert with Its Upcycling Approach
by: Liao, Ning, et al.
Published: (2026)

PonderLM: Pretraining Language Models to Ponder in Continuous Space
by: Zeng, Boyi, et al.
Published: (2025)

PonderLM-2: Pretraining LLM with Latent Thoughts in Continuous Space
by: Zeng, Boyi, et al.
Published: (2025)

NTKMTL: Mitigating Task Imbalance in Multi-Task Learning from Neural Tangent Kernel Perspective
by: Qin, Xiaohan, et al.
Published: (2025)

Boosting Order-Preserving and Transferability for Neural Architecture Search: a Joint Architecture Refined Search and Fine-tuning Approach
by: Zhang, Beichen, et al.
Published: (2024)

SparseX: Efficient Segment-Level KV Cache Sharing for Interleaved LLM Serving
by: Zhang, Quqing, et al.
Published: (2026)

When Autonomy Goes Rogue: Preparing for Risks of Multi-Agent Collusion in Social Systems
by: Ren, Qibing, et al.
Published: (2025)

Pretraining with Token-Level Adaptive Latent Chain-of-Thought
by: Zeng, Boyi, et al.
Published: (2026)

Exploring Training on Heterogeneous Data with Mixture of Low-rank Adapters
by: Zhou, Yuhang, et al.
Published: (2024)

Mitigating Noisy Correspondence by Geometrical Structure Consistency Learning
by: Zhao, Zihua, et al.
Published: (2024)

Innovator-VL: A Multimodal Large Language Model for Scientific Discovery
by: Wen, Zichen, et al.
Published: (2026)

Dual-granularity Sinkhorn Distillation for Enhanced Learning from Long-tailed Noisy Data
by: Hong, Feng, et al.
Published: (2025)

Differential-informed Sample Selection Accelerates Multimodal Contrastive Learning
by: Zhao, Zihua, et al.
Published: (2025)

MLP Memory: A Retriever-Pretrained Memory for Large Language Models
by: Wei, Rubin, et al.
Published: (2025)

Memory Decoder: A Pretrained, Plug-and-Play Memory for Large Language Models
by: Cao, Jiaqi, et al.
Published: (2025)

A Sanity Check on Composed Image Retrieval
by: Liu, Yikun, et al.
Published: (2026)

FTII-Bench: A Comprehensive Multimodal Benchmark for Flow Text with Image Insertion
by: Ruan, Jiacheng, et al.
Published: (2024)

Llama 3 Meets MoE: Efficient Upcycling
by: Vavre, Aditya, et al.
Published: (2024)

ssToken: Self-modulated and Semantic-aware Token Selection for LLM Fine-tuning
by: Qin, Xiaohan, et al.
Published: (2025)

GraphKV: Breaking the Static Selection Paradigm with Graph-Based KV Cache Eviction
by: Li, Xuelin, et al.
Published: (2025)

PonderLM-3: Adaptive Token-Wise Pondering with Differentiable Masking
by: Li, He, et al.
Published: (2026)

MM-CamObj: A Comprehensive Multimodal Dataset for Camouflaged Object Scenarios
by: Ruan, Jiacheng, et al.
Published: (2024)

Entanglement distillation on symmetric two-qutrit entangled states of rank five
by: Song, Zihua, et al.
Published: (2025)

High-Dimensional Carrier-Assisted Entanglement Purification Based on Mutually Unbiased Bases
by: Song, Zihua, et al.
Published: (2026)

Marco-MoE: Open Multilingual Mixture-of-Expert Language Models with Efficient Upcycling
by: Jiang, Fan, et al.
Published: (2026)

AWM: Accurate Weight-Matrix Fingerprint for Large Language Models
by: Zeng, Boyi, et al.
Published: (2025)

Unveiling the Unusual Electron–Phonon Interaction and Quasi‐Particle Dynamics in type‐II Weyl Semimetal NbIrTe4
by: Kaiwen Sun, et al.
Published: (2024)

AdaPonderLM: Gated Pondering Language Models with Token-Wise Adaptive Depth
by: Song, Shixiang, et al.
Published: (2026)

Towards Controlled Table-to-Text Generation with Scientific Reasoning
by: Guo, Zhixin, et al.
Published: (2023)

Moderator: Moderating Text-to-Image Diffusion Models through Fine-grained Context-based Policies
by: Wang, Peiran, et al.
Published: (2024)

Dirichlet-Prior Shaping: Guiding Expert Specialization in Upcycled MoEs
by: Mirvakhabova, Leyla, et al.
Published: (2025)

G4Seg: Generation for Inexact Segmentation Refinement with Diffusion Models
by: Zhang, Tianjiao, et al.
Published: (2025)

Domain-Inspired Sharpness-Aware Minimization Under Domain Shifts
by: Zhang, Ruipeng, et al.
Published: (2024)

Zero-shot Composed Text-Image Retrieval
by: Liu, Yikun, et al.
Published: (2023)

MuteSwap: Visual-informed Silent Video Identity Conversion
by: Liu, Yifan, et al.
Published: (2025)

DiVISe: Direct Visual-Input Speech Synthesis Preserving Speaker Characteristics And Intelligibility
by: Liu, Yifan, et al.
Published: (2025)

Deep Reprogramming Distillation for Medical Foundation Models
by: Du, Siyuan, et al.
Published: (2026)

How Does Environmental Information Disclosure Affect Corporate Environmental Performance? Evidence from Chinese A-Share Listed Companies
by: Lin, Zehao
Published: (2025)

QAEncoder: Towards Aligned Representation Learning in Question Answering Systems
by: Wang, Zhengren, et al.
Published: (2024)

AutoAgent: Evolving Cognition and Elastic Memory Orchestration for Adaptive Agents
by: Wang, Xiaoxing, et al.
Published: (2026)