Cover Image

Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Experts

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Wang, Lean, Gao, Huazuo, Zhao, Chenggang, Sun, Xu, Dai, Damai
Format:	Preprint
Published:	2024
Subjects:	Machine Learning Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Coupling Experts and Routers in Mixture-of-Experts via an Auxiliary Loss
by: Lv, Ang, et al.
Published: (2025)

Demons in the Detail: On Implementing Load Balancing Loss for Training Specialized Mixture-of-Expert Models
by: Qiu, Zihan, et al.
Published: (2025)

Binary-Integer-Programming Based Algorithm for Expert Load Balancing in Mixture-of-Experts Models
by: Sun, Yuan
Published: (2025)

Latent Prototype Routing: Achieving Near-Perfect Load Balancing in Mixture-of-Experts
by: Yang, Jiajie
Published: (2025)

Let the Expert Stick to His Last: Expert-Specialized Fine-Tuning for Sparse Architectural Large Language Models
by: Wang, Zihan, et al.
Published: (2024)

Exploring Activation Patterns of Parameters in Language Models
by: Wang, Yudong, et al.
Published: (2024)

mHC: Manifold-Constrained Hyper-Connections
by: Xie, Zhenda, et al.
Published: (2025)

Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention
by: Yuan, Jingyang, et al.
Published: (2025)

DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models
by: Dai, Damai, et al.
Published: (2024)

Diversifying the Expert Knowledge for Task-Agnostic Pruning in Sparse Mixture-of-Experts
by: Zhang, Zeliang, et al.
Published: (2024)

Routing-Free Mixture-of-Experts
by: Liu, Yilun, et al.
Published: (2026)

GatePro: Parameter-Free Expert Selection Optimization for Mixture-of-Experts Models
by: Zheng, Chen, et al.
Published: (2025)

HMoE: Heterogeneous Mixture of Experts for Language Modeling
by: Wang, An, et al.
Published: (2024)

A Theoretical Framework for Auxiliary-Loss-Free Load Balancing of Sparse Mixture-of-Experts in Large-Scale AI Models
by: Han, X. Y., et al.
Published: (2025)

Maximum Score Routing For Mixture-of-Experts
by: Dong, Bowen, et al.
Published: (2025)

Your Mixture-of-Experts LLM Is Secretly an Embedding Model For Free
by: Li, Ziyue, et al.
Published: (2024)

Mixture of Lookup Experts
by: Jie, Shibo, et al.
Published: (2025)

SEER-MoE: Sparse Expert Efficiency through Regularization for Mixture-of-Experts
by: Muzio, Alexandre, et al.
Published: (2024)

Mixture of Weight-shared Heterogeneous Group Attention Experts for Dynamic Token-wise KV Optimization
by: Song, Guanghui, et al.
Published: (2025)

Towards a Comprehensive Scaling Law of Mixture-of-Experts
by: Zhao, Guoliang, et al.
Published: (2025)

Preserving Long-Tailed Expert Information in Mixture-of-Experts Tuning
by: He, Haoze, et al.
Published: (2026)

Chain-of-Experts: Unlocking the Communication Power of Mixture-of-Experts Models
by: Wang, Zihan, et al.
Published: (2025)

Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts
by: Wang, Haoxiang, et al.
Published: (2024)

Mixture Compressor for Mixture-of-Experts LLMs Gains More
by: Huang, Wei, et al.
Published: (2024)

DECO: Sparse Mixture-of-Experts with Dense-Comparable Performance on End-Side Devices
by: Song, Chenyang, et al.
Published: (2026)

BlockFFN: Towards End-Side Acceleration-Friendly Mixture-of-Experts with Chunk-Level Activation Sparsity
by: Song, Chenyang, et al.
Published: (2025)

When Are Experts Misrouted? Counterfactual Routing Analysis in Mixture-of-Experts Language Models
by: Yoon, Youngsik, et al.
Published: (2026)

$\infty$-MoE: Generalizing Mixture of Experts to Infinite Experts
by: Takashiro, Shota, et al.
Published: (2026)

Beyond Benchmarks: Understanding Mixture-of-Experts Models through Internal Mechanisms
by: Ying, Jiahao, et al.
Published: (2025)

Mixture of Experts Made Intrinsically Interpretable
by: Yang, Xingyi, et al.
Published: (2025)

Optimizing Pre-Training Data Mixtures with Mixtures of Data Expert Models
by: Belenki, Lior, et al.
Published: (2025)

A Survey on Mixture of Experts in Large Language Models
by: Cai, Weilin, et al.
Published: (2024)

Multi-Head Mixture-of-Experts
by: Wu, Xun, et al.
Published: (2024)

Domain-Specific Pruning of Large Mixture-of-Experts Models with Few-shot Demonstrations
by: Dong, Zican, et al.
Published: (2025)

Routers Learn the Geometry of Their Experts: Geometric Coupling in Sparse Mixture-of-Experts
by: Ahrac, Sagi, et al.
Published: (2026)

Mixture of LoRA Experts
by: Wu, Xun, et al.
Published: (2024)

Bayesian Mixture of Experts For Large Language Models
by: Dialameh, Maryam, et al.
Published: (2025)

MoFE: Mixture of Frozen Experts Architecture
by: Seo, Jean, et al.
Published: (2025)

Slicing and Dicing: Configuring Optimal Mixtures of Experts
by: Li, Margaret, et al.
Published: (2026)

A Closer Look into Mixture-of-Experts in Large Language Models
by: Lo, Ka Man, et al.
Published: (2024)