Symmetry-Compatible Principle for Optimizer Design: Embeddings, LM Heads, SwiGLU MLPs, and MoE Routers

Documenti analoghi

• PolarGrad: A Class of Matrix-Gradient Optimizers from a Unifying Preconditioning Perspective
  di: Lau, Tim Tsz-Kit, et al.
  Pubblicazione: (2025)
• AudioMAE++: learning better masked audio representations with SwiGLU FFNs
  di: Yadav, Sarthak, et al.
  Pubblicazione: (2025)
• AdAdaGrad: Adaptive Batch Size Schemes for Adaptive Gradient Methods
  di: Lau, Tim Tsz-Kit, et al.
  Pubblicazione: (2024)
• Uncovering Symmetry Transfer in Large Language Models via Layer-Peeled Optimization
  di: Du, Zhehang, et al.
  Pubblicazione: (2026)
• Depth Registers Unlock W4A4 on SwiGLU: A Reader/Generator Decomposition
  di: Liu, Ziyang
  Pubblicazione: (2026)