How to Scale Mixture-of-Experts: From muP to the Maximally Scale-Stable Parameterization

Documenti analoghi

• On the Surprising Effectiveness of Large Learning Rates under Standard Width Scaling
  di: Haas, Moritz, et al.
  Pubblicazione: (2025)
• μP$^2$: Effective Sharpness Aware Minimization Requires Layerwise Perturbation Scaling
  di: Haas, Moritz, et al.
  Pubblicazione: (2024)
• Explaining Kernel Clustering via Decision Trees
  di: Fleissner, Maximilian, et al.
  Pubblicazione: (2024)
• Training Neural Networks at Any Scale
  di: Pethick, Thomas, et al.
  Pubblicazione: (2025)
• Weight Decay may matter more than muP for Learning Rate Transfer in Practice
  di: Kosson, Atli, et al.
  Pubblicazione: (2025)