Copertina

A Closer Look into Mixture-of-Experts in Large Language Models

Fuente: arXiv

Salvato in:

Dettagli Bibliografici
Autori principali:	Lo, Ka Man, Huang, Zeyu, Qiu, Zihan, Wang, Zili, Fu, Jie
Natura:	Preprint
Pubblicazione:	2024
Soggetti:	Computation and Language Machine Learning
Accesso online:	Acceder al recurso
Tags:	Aggiungi Tag Nessun Tag, puoi essere il primo ad aggiungerne!!

Documenti analoghi

Layerwise Recurrent Router for Mixture-of-Experts
di: Qiu, Zihan, et al.
Pubblicazione: (2024)

A Closer Look at Machine Unlearning for Large Language Models
di: Yuan, Xiaojian, et al.
Pubblicazione: (2024)

A Survey on Mixture of Experts in Large Language Models
di: Cai, Weilin, et al.
Pubblicazione: (2024)

Demons in the Detail: On Implementing Load Balancing Loss for Training Specialized Mixture-of-Expert Models
di: Qiu, Zihan, et al.
Pubblicazione: (2025)

Unlocking Continual Learning Abilities in Language Models
di: Du, Wenyu, et al.
Pubblicazione: (2024)

Bayesian Mixture of Experts For Large Language Models
di: Dialameh, Maryam, et al.
Pubblicazione: (2025)

ToMoE: Converting Dense Large Language Models to Mixture-of-Experts through Dynamic Structural Pruning
di: Gao, Shangqian, et al.
Pubblicazione: (2025)

Chain-of-Experts: Unlocking the Communication Power of Mixture-of-Experts Models
di: Wang, Zihan, et al.
Pubblicazione: (2025)

Stacking Your Transformers: A Closer Look at Model Growth for Efficient LLM Pre-Training
di: Du, Wenyu, et al.
Pubblicazione: (2024)

Probing Semantic Routing in Large Mixture-of-Expert Models
di: Olson, Matthew Lyle, et al.
Pubblicazione: (2025)

Upcycling Large Language Models into Mixture of Experts
di: He, Ethan, et al.
Pubblicazione: (2024)

Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Models
di: Lu, Xudong, et al.
Pubblicazione: (2024)

Unlocking Emergent Modularity in Large Language Models
di: Qiu, Zihan, et al.
Pubblicazione: (2023)

When Are Experts Misrouted? Counterfactual Routing Analysis in Mixture-of-Experts Language Models
di: Yoon, Youngsik, et al.
Pubblicazione: (2026)

MoLAE: Mixture of Latent Experts for Parameter-Efficient Language Models
di: Liu, Zehua, et al.
Pubblicazione: (2025)

HMoE: Heterogeneous Mixture of Experts for Language Modeling
di: Wang, An, et al.
Pubblicazione: (2024)

Mixture of Lookup Experts
di: Jie, Shibo, et al.
Pubblicazione: (2025)

HyperMoE: Towards Better Mixture of Experts via Transferring Among Experts
di: Zhao, Hao, et al.
Pubblicazione: (2024)

Blending Supervised and Reinforcement Fine-Tuning with Prefix Sampling
di: Huang, Zeyu, et al.
Pubblicazione: (2025)

Scaling Laws for Upcycling Mixture-of-Experts Language Models
di: Liew, Seng Pei, et al.
Pubblicazione: (2025)

Mixture of Heterogeneous Grouped Experts for Language Modeling
di: Ma, Zhicheng, et al.
Pubblicazione: (2026)

OLMoE: Open Mixture-of-Experts Language Models
di: Muennighoff, Niklas, et al.
Pubblicazione: (2024)

A Closer Look at Classification Evaluation Metrics and a Critical Reflection of Common Evaluation Practice
di: Opitz, Juri
Pubblicazione: (2024)

Closer Look at Efficient Inference Methods: A Survey of Speculative Decoding
di: Ryu, Hyun, et al.
Pubblicazione: (2024)

Domain-Specific Pruning of Large Mixture-of-Experts Models with Few-shot Demonstrations
di: Dong, Zican, et al.
Pubblicazione: (2025)

Mixtures of SubExperts for Large Language Continual Learning
di: Kang, Haeyong
Pubblicazione: (2025)

Let the Expert Stick to His Last: Expert-Specialized Fine-Tuning for Sparse Architectural Large Language Models
di: Wang, Zihan, et al.
Pubblicazione: (2024)

LIBMoE: A Library for comprehensive benchmarking Mixture of Experts in Large Language Models
di: Nguyen, Nam V., et al.
Pubblicazione: (2024)

FactorLLM: Factorizing Knowledge via Mixture of Experts for Large Language Models
di: Zhao, Zhongyu, et al.
Pubblicazione: (2024)

MoSE: Mixture of Slimmable Experts for Efficient and Adaptive Language Models
di: Tastan, Nurbek, et al.
Pubblicazione: (2026)

Mixture Compressor for Mixture-of-Experts LLMs Gains More
di: Huang, Wei, et al.
Pubblicazione: (2024)

Mixture of LoRA Experts
di: Wu, Xun, et al.
Pubblicazione: (2024)

Multi-Head Mixture-of-Experts
di: Wu, Xun, et al.
Pubblicazione: (2024)

Lory: Fully Differentiable Mixture-of-Experts for Autoregressive Language Model Pre-training
di: Zhong, Zexuan, et al.
Pubblicazione: (2024)

Optimizing Pre-Training Data Mixtures with Mixtures of Data Expert Models
di: Belenki, Lior, et al.
Pubblicazione: (2025)

Pruning and Distilling Mixture-of-Experts into Dense Language Models
di: Kim, Junhyuck, et al.
Pubblicazione: (2026)

FourierMoE: Fourier Mixture-of-Experts Adaptation of Large Language Models
di: Jiang, Juyong, et al.
Pubblicazione: (2026)

MLP Fusion: Towards Efficient Fine-tuning of Dense and Mixture-of-Experts Language Models
di: Ai, Mengting, et al.
Pubblicazione: (2023)

Towards Principled Design of Mixture-of-Experts Language Models under Memory and Inference Constraints
di: Liew, Seng Pei, et al.
Pubblicazione: (2026)

GatePro: Parameter-Free Expert Selection Optimization for Mixture-of-Experts Models
di: Zheng, Chen, et al.
Pubblicazione: (2025)