TEON: Tensorized Orthonormalization Beyond Layer-Wise Muon for Large Language Model Pre-Training

Documenti analoghi

• Muon$^2$: Boosting Muon via Adaptive Second-Moment Preconditioning
  di: Liu, Ziyue, et al.
  Pubblicazione: (2026)
• MuonQ: Enhancing Low-Bit Muon Quantization via Directional Fidelity Optimization
  di: Su, Yupeng, et al.
  Pubblicazione: (2026)
• MUON+: Towards More Effective Muon via One Additional Normalization Step for LLM Pre-training
  di: Zhang, Ruijie, et al.
  Pubblicazione: (2026)
• LaX: Boosting Low-Rank Training of Foundation Models via Latent Crossing
  di: Zhang, Ruijie, et al.
  Pubblicazione: (2025)
• CoLA: Compute-Efficient Pre-Training of LLMs via Low-Rank Activation
  di: Liu, Ziyue, et al.
  Pubblicazione: (2025)