Spectra: Rethinking Optimizers for LLMs Under Spectral Anisotropy
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2026
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866910019750985728 |
|---|---|
| author | Huang, Zhendong Cao, Hengjie Dong, Fang Huang, Ruijun Chen, Mengyi Yang, Yifeng Zhang, Xin Chen, Anrui Dong, Mingzhi Wang, Yujiang Hou, Jinlong Lv, Qin Dick, Robert P. Cheng, Yuan Yang, Fan Lu, Tun Shang, Li |
| author_facet | Huang, Zhendong Cao, Hengjie Dong, Fang Huang, Ruijun Chen, Mengyi Yang, Yifeng Zhang, Xin Chen, Anrui Dong, Mingzhi Wang, Yujiang Hou, Jinlong Lv, Qin Dick, Robert P. Cheng, Yuan Yang, Fan Lu, Tun Shang, Li |
| contents | Gradient signals in LLM training are highly anisotropic: recurrent linguistic structure concentrates energy into a small set of dominant spectral directions, while context specific information resides in a long tail. We show that this spike tail separation persists throughout training, with the spike occupying only about 1.5% of directions yet dominating optimizer statistics. This dominance suppresses tail learning by contracting tail updates through second moment normalization and tightening the globally stable learning rate bound. Motivated by this analysis, we propose Spectra, a spike aware optimizer that suppresses the dominant low rank spike subspace without amplifying the noise sensitive spectral tail. Spectra tracks the spike subspace via cached, warm started power iteration and applies low rank spectral shaping with negligible overhead and substantially reduced optimizer state memory. On LLaMA3 8B trained on 50B tokens, Spectra reaches the same target loss 30% faster than AdamW, reduces per step end to end overhead by 0.7%, cuts optimizer state memory by 49.25%, and improves average downstream accuracy by 1.62%. Compared to Muon, Spectra is 5.1x faster in optimizer processing time, achieves a lower final loss, and improves average accuracy by 0.66%. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_11185 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Spectra: Rethinking Optimizers for LLMs Under Spectral Anisotropy Huang, Zhendong Cao, Hengjie Dong, Fang Huang, Ruijun Chen, Mengyi Yang, Yifeng Zhang, Xin Chen, Anrui Dong, Mingzhi Wang, Yujiang Hou, Jinlong Lv, Qin Dick, Robert P. Cheng, Yuan Yang, Fan Lu, Tun Shang, Li Machine Learning Artificial Intelligence Gradient signals in LLM training are highly anisotropic: recurrent linguistic structure concentrates energy into a small set of dominant spectral directions, while context specific information resides in a long tail. We show that this spike tail separation persists throughout training, with the spike occupying only about 1.5% of directions yet dominating optimizer statistics. This dominance suppresses tail learning by contracting tail updates through second moment normalization and tightening the globally stable learning rate bound. Motivated by this analysis, we propose Spectra, a spike aware optimizer that suppresses the dominant low rank spike subspace without amplifying the noise sensitive spectral tail. Spectra tracks the spike subspace via cached, warm started power iteration and applies low rank spectral shaping with negligible overhead and substantially reduced optimizer state memory. On LLaMA3 8B trained on 50B tokens, Spectra reaches the same target loss 30% faster than AdamW, reduces per step end to end overhead by 0.7%, cuts optimizer state memory by 49.25%, and improves average downstream accuracy by 1.62%. Compared to Muon, Spectra is 5.1x faster in optimizer processing time, achieves a lower final loss, and improves average accuracy by 0.66%. |
| title | Spectra: Rethinking Optimizers for LLMs Under Spectral Anisotropy |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2602.11185 |