Demons in the Detail: On Implementing Load Balancing Loss for Training Specialized Mixture-of-Expert Models

Documenti analoghi

• Gated Attention for Large Language Models: Non-linearity, Sparsity, and Attention-Sink-Free
  di: Qiu, Zihan, et al.
  Pubblicazione: (2025)
• A Unified View of Attention and Residual Sinks: Outlier-Driven Rescaling is Essential for Transformer Training
  di: Qiu, Zihan, et al.
  Pubblicazione: (2026)
• Layerwise Recurrent Router for Mixture-of-Experts
  di: Qiu, Zihan, et al.
  Pubblicazione: (2024)
• Canzona: A Unified, Asynchronous, and Load-Balanced Framework for Distributed Matrix-based Optimizers
  di: Wang, Liangyu, et al.
  Pubblicazione: (2026)
• Auxiliary-Loss-Free Load Balancing Strategy for Mixture-of-Experts
  di: Wang, Lean, et al.
  Pubblicazione: (2024)