Image de couverture de livre

Scaling Laws for Upcycling Mixture-of-Experts Language Models

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Liew, Seng Pei, Kato, Takuya, Takase, Sho
Format:	Preprint
Publié:	2025
Sujets:	Machine Learning Computation and Language
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

Reusing Overtrained Language Models Saturates Scaling
par: Liew, Seng Pei, et autres
Publié: (2025)

Towards Principled Design of Mixture-of-Experts Language Models under Memory and Inference Constraints
par: Liew, Seng Pei, et autres
Publié: (2026)

Upcycling Large Language Models into Mixture of Experts
par: He, Ethan, et autres
Publié: (2024)

Drop-Upcycling: Training Sparse Mixture of Experts with Partial Re-initialization
par: Nakamura, Taishi, et autres
Publié: (2025)

MoIN: Mixture of Introvert Experts to Upcycle an LLM
par: Tejankar, Ajinkya, et autres
Publié: (2024)

Training-Free Dynamic Upcycling of Expert Language Models
par: Fanì, Eros, et autres
Publié: (2026)

Scaling Laws for Fine-Grained Mixture of Experts
par: Krajewski, Jakub, et autres
Publié: (2024)

Towards a Comprehensive Scaling Law of Mixture-of-Experts
par: Zhao, Guoliang, et autres
Publié: (2025)

Large Vocabulary Size Improves Large Language Models
par: Takase, Sho, et autres
Publié: (2024)

Upcycling Instruction Tuning from Dense to Mixture-of-Experts via Parameter Merging
par: Hui, Tingfeng, et autres
Publié: (2024)

MoLEx: Mixture of Layer Experts for Finetuning with Sparse Upcycling
par: Teo, Rachel S. Y., et autres
Publié: (2025)

Bayesian Mixture of Experts For Large Language Models
par: Dialameh, Maryam, et autres
Publié: (2025)

Joint MoE Scaling Laws: Mixture of Experts Can Be Memory Efficient
par: Ludziejewski, Jan, et autres
Publié: (2025)

Revisiting the Capacity Gap in Chain-of-Thought Distillation from a Practical Perspective
par: Kajitsuka, Tokio, et autres
Publié: (2026)

A Survey on Mixture of Experts in Large Language Models
par: Cai, Weilin, et autres
Publié: (2024)

HMoE: Heterogeneous Mixture of Experts for Language Modeling
par: Wang, An, et autres
Publié: (2024)

CMR Scaling Law: Predicting Critical Mixture Ratios for Continual Pre-training of Language Models
par: Gu, Jiawei, et autres
Publié: (2024)

When Are Experts Misrouted? Counterfactual Routing Analysis in Mixture-of-Experts Language Models
par: Yoon, Youngsik, et autres
Publié: (2026)

A Closer Look into Mixture-of-Experts in Large Language Models
par: Lo, Ka Man, et autres
Publié: (2024)

Scaling Laws for Mixture Pretraining Under Data Constraints
par: Sedova, Anastasiia, et autres
Publié: (2026)

Mixture of Heterogeneous Grouped Experts for Language Modeling
par: Ma, Zhicheng, et autres
Publié: (2026)

MoLAE: Mixture of Latent Experts for Parameter-Efficient Language Models
par: Liu, Zehua, et autres
Publié: (2025)

XFT: Unlocking the Power of Code Instruction Tuning by Simply Merging Upcycled Mixture-of-Experts
par: Ding, Yifeng, et autres
Publié: (2024)

MoSE: Mixture of Slimmable Experts for Efficient and Adaptive Language Models
par: Tastan, Nurbek, et autres
Publié: (2026)

Parallel Scaling Law for Language Models
par: Chen, Mouxiang, et autres
Publié: (2025)

Scaling Laws for Multilingual Language Models
par: He, Yifei, et autres
Publié: (2024)

Scaling Embeddings Outperforms Scaling Experts in Language Models
par: Liu, Hong, et autres
Publié: (2026)

Chain-of-Experts: Unlocking the Communication Power of Mixture-of-Experts Models
par: Wang, Zihan, et autres
Publié: (2025)

Pre-training LLM without Learning Rate Decay Enhances Supervised Fine-Tuning
par: Yano, Kazuki, et autres
Publié: (2026)

Lory: Fully Differentiable Mixture-of-Experts for Autoregressive Language Model Pre-training
par: Zhong, Zexuan, et autres
Publié: (2024)

Optimizing Pre-Training Data Mixtures with Mixtures of Data Expert Models
par: Belenki, Lior, et autres
Publié: (2025)

Mixture of Lookup Experts
par: Jie, Shibo, et autres
Publié: (2025)

Pruning and Distilling Mixture-of-Experts into Dense Language Models
par: Kim, Junhyuck, et autres
Publié: (2026)

OLMoE: Open Mixture-of-Experts Language Models
par: Muennighoff, Niklas, et autres
Publié: (2024)

Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Models
par: Lu, Xudong, et autres
Publié: (2024)

MLP Fusion: Towards Efficient Fine-tuning of Dense and Mixture-of-Experts Language Models
par: Ai, Mengting, et autres
Publié: (2023)

GatePro: Parameter-Free Expert Selection Optimization for Mixture-of-Experts Models
par: Zheng, Chen, et autres
Publié: (2025)

Every Expert Matters: Towards Effective Knowledge Distillation for Mixture-of-Experts Language Models
par: Kim, Gyeongman, et autres
Publié: (2025)

Scaling Laws for Discriminative Classification in Large Language Models
par: Wyatte, Dean, et autres
Publié: (2024)

Mixture Compressor for Mixture-of-Experts LLMs Gains More
par: Huang, Wei, et autres
Publié: (2024)