Image de couverture de livre

Drop-Upcycling: Training Sparse Mixture of Experts with Partial Re-initialization

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Nakamura, Taishi, Akiba, Takuya, Fujii, Kazuki, Oda, Yusuke, Yokota, Rio, Suzuki, Jun
Format:	Preprint
Publié:	2025
Sujets:	Computation and Language Artificial Intelligence Machine Learning
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

Optimal Sparsity of Mixture-of-Experts Language Models for Reasoning Tasks
par: Nakamura, Taishi, et autres
Publié: (2025)

On the Optimal Reasoning Length for RL-Trained Language Models
par: Nohara, Daisuke, et autres
Publié: (2026)

Balancing Speed and Stability: The Trade-offs of FP8 vs. BF16 Training in LLMs
par: Fujii, Kazuki, et autres
Publié: (2024)

Continual Pre-Training for Cross-Lingual LLM Adaptation: Enhancing Japanese Language Capabilities
par: Fujii, Kazuki, et autres
Publié: (2024)

MoIN: Mixture of Introvert Experts to Upcycle an LLM
par: Tejankar, Ajinkya, et autres
Publié: (2024)

Upcycling Large Language Models into Mixture of Experts
par: He, Ethan, et autres
Publié: (2024)

MoLEx: Mixture of Layer Experts for Finetuning with Sparse Upcycling
par: Teo, Rachel S. Y., et autres
Publié: (2025)

Agent Skill Acquisition for Large Language Models via CycleQD
par: Kuroki, So, et autres
Publié: (2024)

Dense Training, Sparse Inference: Rethinking Training of Mixture-of-Experts Language Models
par: Pan, Bowen, et autres
Publié: (2024)

Building a Large Japanese Web Corpus for Large Language Models
par: Okazaki, Naoaki, et autres
Publié: (2024)

Upcycling Instruction Tuning from Dense to Mixture-of-Experts via Parameter Merging
par: Hui, Tingfeng, et autres
Publié: (2024)

Extending the Context of Pretrained LLMs by Dropping Their Positional Embeddings
par: Gelberg, Yoav, et autres
Publié: (2025)

XFT: Unlocking the Power of Code Instruction Tuning by Simply Merging Upcycled Mixture-of-Experts
par: Ding, Yifeng, et autres
Publié: (2024)

Expert Upcycling: Shifting the Compute-Efficient Frontier of Mixture-of-Experts
par: Dwivedi, Chaitanya, et autres
Publié: (2026)

Nexus: Specialization meets Adaptability for Efficiently Training Mixture of Experts
par: Gritsch, Nikolas, et autres
Publié: (2024)

HodgeCover: Higher-Order Topological Coverage Drives Compression of Sparse Mixture-of-Experts
par: Zhong, Tao, et autres
Publié: (2026)

Training Sparse Mixture Of Experts Text Embedding Models
par: Nussbaum, Zach, et autres
Publié: (2025)

Rewriting Pre-Training Data Boosts LLM Performance in Math and Code
par: Fujii, Kazuki, et autres
Publié: (2025)

QuantMoE-Bench: Examining Post-Training Quantization for Mixture-of-Experts
par: Li, Pingzhi, et autres
Publié: (2024)

Exploring Expert Specialization through Unsupervised Training in Sparse Mixture of Experts
par: Nikolic, Strahinja, et autres
Publié: (2025)

Scaling Laws for Upcycling Mixture-of-Experts Language Models
par: Liew, Seng Pei, et autres
Publié: (2025)

Dense Backpropagation Improves Training for Sparse Mixture-of-Experts
par: Panda, Ashwinee, et autres
Publié: (2025)

Marco-MoE: Open Multilingual Mixture-of-Expert Language Models with Efficient Upcycling
par: Jiang, Fan, et autres
Publié: (2026)

Routing-Free Mixture-of-Experts
par: Liu, Yilun, et autres
Publié: (2026)

Multi-Head Mixture-of-Experts
par: Wu, Xun, et autres
Publié: (2024)

Multilingual Routing in Mixture-of-Experts
par: Bandarkar, Lucas, et autres
Publié: (2025)

SEUF: Is Unlearning One Expert Enough for Mixture-of-Experts LLMs?
par: Zhuang, Haomin, et autres
Publié: (2024)

MomentumSMoE: Integrating Momentum into Sparse Mixture of Experts
par: Teo, Rachel S. Y., et autres
Publié: (2024)

On the Spatial Structure of Mixture-of-Experts in Transformers
par: Bershatsky, Daniel, et autres
Publié: (2025)

Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Models
par: Lu, Xudong, et autres
Publié: (2024)

Refactoring Programs Using Large Language Models with Few-Shot Examples
par: Shirafuji, Atsushi, et autres
Publié: (2023)

Mixture of Heterogeneous Grouped Experts for Language Modeling
par: Ma, Zhicheng, et autres
Publié: (2026)

Scaling Laws for Fine-Grained Mixture of Experts
par: Krajewski, Jakub, et autres
Publié: (2024)

Dynamic Experts Search: Enhancing Reasoning in Mixture-of-Experts LLMs at Test Time
par: Han, Yixuan, et autres
Publié: (2025)

Parameter-Efficient Routed Fine-Tuning: Mixture-of-Experts Demands Mixture of Adaptation Modules
par: Liu, Yilun, et autres
Publié: (2025)

Every Expert Matters: Towards Effective Knowledge Distillation for Mixture-of-Experts Language Models
par: Kim, Gyeongman, et autres
Publié: (2025)

TAID: Temporally Adaptive Interpolated Distillation for Efficient Knowledge Transfer in Language Models
par: Shing, Makoto, et autres
Publié: (2025)

Mixtures of SubExperts for Large Language Continual Learning
par: Kang, Haeyong
Publié: (2025)

MobileMoE: Scaling On-Device Mixture of Experts
par: Chen, Yanbei, et autres
Publié: (2026)

Pruning and Distilling Mixture-of-Experts into Dense Language Models
par: Kim, Junhyuck, et autres
Publié: (2026)