Guardado en:
| Autores principales: | , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2026
|
| Materias: | |
| Acceso en línea: | https://arxiv.org/abs/2601.08393 |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866915834866171904 |
|---|---|
| author | Xie, Tian Luo, Haoming Tang, Haoyu Hu, Yiwen Liu, Jason Klein Ren, Qingnan Wang, Yang Zhao, Wayne Xin Yan, Rui Su, Bing Luo, Chong Guo, Baining |
| author_facet | Xie, Tian Luo, Haoming Tang, Haoyu Hu, Yiwen Liu, Jason Klein Ren, Qingnan Wang, Yang Zhao, Wayne Xin Yan, Rui Su, Bing Luo, Chong Guo, Baining |
| contents | Scaling large models requires optimization strategies that ensure rapid convergence grounded in stability. Maximal Update Parametrization ($\boldsymbolμ$P) provides a theoretical safeguard for width-invariant $Θ(1)$ activation control, whereas emerging optimizers like Muon are only ``half-aligned'' with these constraints: they control updates but allow weights to drift. To address this limitation, we introduce the \textbf{Spectral Sphere Optimizer (SSO)}, which enforces strict module-wise spectral constraints on both weights and their updates. By deriving the steepest descent direction on the spectral sphere, SSO realizes a fully $\boldsymbolμ$P-aligned optimization process. To enable large-scale training, we implement SSO as an efficient parallel algorithm within Megatron. Through extensive pretraining on diverse architectures, including Dense 1.7B, MoE 8B-A1B, and 200-layer DeepNet models, SSO consistently outperforms AdamW and Muon. Furthermore, we observe significant practical stability benefits, including improved MoE router load balancing, suppressed outliers, and strictly bounded activations. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_08393 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Controlled LLM Training on Spectral Sphere Xie, Tian Luo, Haoming Tang, Haoyu Hu, Yiwen Liu, Jason Klein Ren, Qingnan Wang, Yang Zhao, Wayne Xin Yan, Rui Su, Bing Luo, Chong Guo, Baining Machine Learning Artificial Intelligence Scaling large models requires optimization strategies that ensure rapid convergence grounded in stability. Maximal Update Parametrization ($\boldsymbolμ$P) provides a theoretical safeguard for width-invariant $Θ(1)$ activation control, whereas emerging optimizers like Muon are only ``half-aligned'' with these constraints: they control updates but allow weights to drift. To address this limitation, we introduce the \textbf{Spectral Sphere Optimizer (SSO)}, which enforces strict module-wise spectral constraints on both weights and their updates. By deriving the steepest descent direction on the spectral sphere, SSO realizes a fully $\boldsymbolμ$P-aligned optimization process. To enable large-scale training, we implement SSO as an efficient parallel algorithm within Megatron. Through extensive pretraining on diverse architectures, including Dense 1.7B, MoE 8B-A1B, and 200-layer DeepNet models, SSO consistently outperforms AdamW and Muon. Furthermore, we observe significant practical stability benefits, including improved MoE router load balancing, suppressed outliers, and strictly bounded activations. |
| title | Controlled LLM Training on Spectral Sphere |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2601.08393 |