Image de couverture de livre

A Survey on Mixture of Experts in Large Language Models

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Cai, Weilin, Jiang, Juyong, Wang, Fan, Tang, Jing, Kim, Sunghun, Huang, Jiayi
Format:	Preprint
Publié:	2024
Sujets:	Machine Learning Computation and Language
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

Shortcut-connected Expert Parallelism for Accelerating Mixture-of-Experts
par: Cai, Weilin, et autres
Publié: (2024)

FourierMoE: Fourier Mixture-of-Experts Adaptation of Large Language Models
par: Jiang, Juyong, et autres
Publié: (2026)

KaSA: Knowledge-Aware Singular-Value Adaptation of Large Language Models
par: Wang, Fan, et autres
Publié: (2024)

Capacity-Aware Inference: Mitigating the Straggler Effect in Mixture of Experts
par: He, Shwai, et autres
Publié: (2025)

WebGen-R1: Incentivizing Large Language Models to Generate Functional and Aesthetic Websites with Reinforcement Learning
par: Jiang, Juyong, et autres
Publié: (2026)

ReflexiCoder: Teaching Large Language Models to Self-Reflect on Generated Code and Self-Correct It via Reinforcement Learning
par: Jiang, Juyong, et autres
Publié: (2026)

A Survey on Large Language Models for Code Generation
par: Jiang, Juyong, et autres
Publié: (2024)

A Closer Look into Mixture-of-Experts in Large Language Models
par: Lo, Ka Man, et autres
Publié: (2024)

MoC-System: Efficient Fault Tolerance for Sparse Mixture-of-Experts Model Training
par: Cai, Weilin, et autres
Publié: (2024)

TAROT: Test-driven and Capability-adaptive Curriculum Reinforcement Fine-tuning for Code Generation with Large Language Models
par: Park, Chansung, et autres
Publié: (2026)

Bayesian Mixture of Experts For Large Language Models
par: Dialameh, Maryam, et autres
Publié: (2025)

Upcycling Large Language Models into Mixture of Experts
par: He, Ethan, et autres
Publié: (2024)

Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Models
par: Lu, Xudong, et autres
Publié: (2024)

Mixture of Lookup Experts
par: Jie, Shibo, et autres
Publié: (2025)

Pruning and Distilling Mixture-of-Experts into Dense Language Models
par: Kim, Junhyuck, et autres
Publié: (2026)

GatePro: Parameter-Free Expert Selection Optimization for Mixture-of-Experts Models
par: Zheng, Chen, et autres
Publié: (2025)

When Are Experts Misrouted? Counterfactual Routing Analysis in Mixture-of-Experts Language Models
par: Yoon, Youngsik, et autres
Publié: (2026)

HMoE: Heterogeneous Mixture of Experts for Language Modeling
par: Wang, An, et autres
Publié: (2024)

DECO: Sparse Mixture-of-Experts with Dense-Comparable Performance on End-Side Devices
par: Song, Chenyang, et autres
Publié: (2026)

Every Expert Matters: Towards Effective Knowledge Distillation for Mixture-of-Experts Language Models
par: Kim, Gyeongman, et autres
Publié: (2025)

Scaling Laws for Upcycling Mixture-of-Experts Language Models
par: Liew, Seng Pei, et autres
Publié: (2025)

Mixture of Heterogeneous Grouped Experts for Language Modeling
par: Ma, Zhicheng, et autres
Publié: (2026)

ToMoE: Converting Dense Large Language Models to Mixture-of-Experts through Dynamic Structural Pruning
par: Gao, Shangqian, et autres
Publié: (2025)

S'MoRE: Structural Mixture of Residual Experts for Parameter-Efficient LLM Fine-tuning
par: Zeng, Hanqing, et autres
Publié: (2025)

Maximum Score Routing For Mixture-of-Experts
par: Dong, Bowen, et autres
Publié: (2025)

Domain-Specific Pruning of Large Mixture-of-Experts Models with Few-shot Demonstrations
par: Dong, Zican, et autres
Publié: (2025)

Chain-of-Experts: Unlocking the Communication Power of Mixture-of-Experts Models
par: Wang, Zihan, et autres
Publié: (2025)

Mixtures of SubExperts for Large Language Continual Learning
par: Kang, Haeyong
Publié: (2025)

LIBMoE: A Library for comprehensive benchmarking Mixture of Experts in Large Language Models
par: Nguyen, Nam V., et autres
Publié: (2024)

Read-ME: Refactorizing LLMs as Router-Decoupled Mixture of Experts with System Co-Design
par: Cai, Ruisi, et autres
Publié: (2024)

FactorLLM: Factorizing Knowledge via Mixture of Experts for Large Language Models
par: Zhao, Zhongyu, et autres
Publié: (2024)

MoFE: Mixture of Frozen Experts Architecture
par: Seo, Jean, et autres
Publié: (2025)

BlockFFN: Towards End-Side Acceleration-Friendly Mixture-of-Experts with Chunk-Level Activation Sparsity
par: Song, Chenyang, et autres
Publié: (2025)

MoSE: Mixture of Slimmable Experts for Efficient and Adaptive Language Models
par: Tastan, Nurbek, et autres
Publié: (2026)

MoLAE: Mixture of Latent Experts for Parameter-Efficient Language Models
par: Liu, Zehua, et autres
Publié: (2025)

Mixture Compressor for Mixture-of-Experts LLMs Gains More
par: Huang, Wei, et autres
Publié: (2024)

ME-Switch: A Memory-Efficient Expert Switching Framework for Large Language Models
par: Liu, Jing, et autres
Publié: (2024)

Mixture of LoRA Experts
par: Wu, Xun, et autres
Publié: (2024)

Preserving Long-Tailed Expert Information in Mixture-of-Experts Tuning
par: He, Haoze, et autres
Publié: (2026)

Multi-Head Mixture-of-Experts
par: Wu, Xun, et autres
Publié: (2024)