Copertina

Towards Principled Design of Mixture-of-Experts Language Models under Memory and Inference Constraints

Fuente: arXiv

Salvato in:

Dettagli Bibliografici
Autori principali:	Liew, Seng Pei, Shinzato, Kenta, Dong, Yuyang
Natura:	Preprint
Pubblicazione:	2026
Soggetti:	Computation and Language Machine Learning
Accesso online:	Acceder al recurso
Tags:	Aggiungi Tag Nessun Tag, puoi essere il primo ad aggiungerne!!

Documenti analoghi

Scaling Laws for Upcycling Mixture-of-Experts Language Models
di: Liew, Seng Pei, et al.
Pubblicazione: (2025)

Reusing Overtrained Language Models Saturates Scaling
di: Liew, Seng Pei, et al.
Pubblicazione: (2025)

Towards Stable and Effective Reinforcement Learning for Mixture-of-Experts
di: Zhang, Di, et al.
Pubblicazione: (2025)

Bayesian Mixture of Experts For Large Language Models
di: Dialameh, Maryam, et al.
Pubblicazione: (2025)

MLP Fusion: Towards Efficient Fine-tuning of Dense and Mixture-of-Experts Language Models
di: Ai, Mengting, et al.
Pubblicazione: (2023)

Every Expert Matters: Towards Effective Knowledge Distillation for Mixture-of-Experts Language Models
di: Kim, Gyeongman, et al.
Pubblicazione: (2025)

Dense Training, Sparse Inference: Rethinking Training of Mixture-of-Experts Language Models
di: Pan, Bowen, et al.
Pubblicazione: (2024)

$\texttt{MoE-RBench}$: Towards Building Reliable Language Models with Sparse Mixture-of-Experts
di: Chen, Guanjie, et al.
Pubblicazione: (2024)

A Survey on Mixture of Experts in Large Language Models
di: Cai, Weilin, et al.
Pubblicazione: (2024)

HMoE: Heterogeneous Mixture of Experts for Language Modeling
di: Wang, An, et al.
Pubblicazione: (2024)

When Are Experts Misrouted? Counterfactual Routing Analysis in Mixture-of-Experts Language Models
di: Yoon, Youngsik, et al.
Pubblicazione: (2026)

A Closer Look into Mixture-of-Experts in Large Language Models
di: Lo, Ka Man, et al.
Pubblicazione: (2024)

Mixture of Heterogeneous Grouped Experts for Language Modeling
di: Ma, Zhicheng, et al.
Pubblicazione: (2026)

Upcycling Large Language Models into Mixture of Experts
di: He, Ethan, et al.
Pubblicazione: (2024)

MoSE: Mixture of Slimmable Experts for Efficient and Adaptive Language Models
di: Tastan, Nurbek, et al.
Pubblicazione: (2026)

MoLAE: Mixture of Latent Experts for Parameter-Efficient Language Models
di: Liu, Zehua, et al.
Pubblicazione: (2025)

Domain-Specific Pruning of Large Mixture-of-Experts Models with Few-shot Demonstrations
di: Dong, Zican, et al.
Pubblicazione: (2025)

Maximum Score Routing For Mixture-of-Experts
di: Dong, Bowen, et al.
Pubblicazione: (2025)

Chain-of-Experts: Unlocking the Communication Power of Mixture-of-Experts Models
di: Wang, Zihan, et al.
Pubblicazione: (2025)

Lory: Fully Differentiable Mixture-of-Experts for Autoregressive Language Model Pre-training
di: Zhong, Zexuan, et al.
Pubblicazione: (2024)

Optimizing Pre-Training Data Mixtures with Mixtures of Data Expert Models
di: Belenki, Lior, et al.
Pubblicazione: (2025)

Pruning and Distilling Mixture-of-Experts into Dense Language Models
di: Kim, Junhyuck, et al.
Pubblicazione: (2026)

OLMoE: Open Mixture-of-Experts Language Models
di: Muennighoff, Niklas, et al.
Pubblicazione: (2024)

Mixture of Lookup Experts
di: Jie, Shibo, et al.
Pubblicazione: (2025)

Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Models
di: Lu, Xudong, et al.
Pubblicazione: (2024)

FactorLLM: Factorizing Knowledge via Mixture of Experts for Large Language Models
di: Zhao, Zhongyu, et al.
Pubblicazione: (2024)

GatePro: Parameter-Free Expert Selection Optimization for Mixture-of-Experts Models
di: Zheng, Chen, et al.
Pubblicazione: (2025)

Capacity-Aware Inference: Mitigating the Straggler Effect in Mixture of Experts
di: He, Shwai, et al.
Pubblicazione: (2025)

Mixture Compressor for Mixture-of-Experts LLMs Gains More
di: Huang, Wei, et al.
Pubblicazione: (2024)

Optimal Sparsity of Mixture-of-Experts Language Models for Reasoning Tasks
di: Nakamura, Taishi, et al.
Pubblicazione: (2025)

Geometry-Preserving Aggregation for Mixture-of-Experts Embedding Models
di: Kachuee, Sajjad, et al.
Pubblicazione: (2026)

Alloc-MoE: Budget-Aware Expert Activation Allocation for Efficient Mixture-of-Experts Inference
di: Liu, Baihui, et al.
Pubblicazione: (2026)

Binary-Integer-Programming Based Algorithm for Expert Load Balancing in Mixture-of-Experts Models
di: Sun, Yuan
Pubblicazione: (2025)

Towards a Comprehensive Scaling Law of Mixture-of-Experts
di: Zhao, Guoliang, et al.
Pubblicazione: (2025)

Elastic MoE: Unlocking the Inference-Time Scalability of Mixture-of-Experts
di: Gu, Naibin, et al.
Pubblicazione: (2025)

$\infty$-MoE: Generalizing Mixture of Experts to Infinite Experts
di: Takashiro, Shota, et al.
Pubblicazione: (2026)

Preserving Long-Tailed Expert Information in Mixture-of-Experts Tuning
di: He, Haoze, et al.
Pubblicazione: (2026)

Coupling Experts and Routers in Mixture-of-Experts via an Auxiliary Loss
di: Lv, Ang, et al.
Pubblicazione: (2025)

Deconstructing Attention: Investigating Design Principles for Effective Language Modeling
di: Xue, Huiyin, et al.
Pubblicazione: (2025)

Your Mixture-of-Experts LLM Is Secretly an Embedding Model For Free
di: Li, Ziyue, et al.
Pubblicazione: (2024)