Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2605.13079 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914562712797184 |
|---|---|
| author | Nguyen, Tien-Phat Nguyen, Truong Truong, Minh-Phuc Nguyen, Tuc Bailey, James Le, Trung |
| author_facet | Nguyen, Tien-Phat Nguyen, Truong Truong, Minh-Phuc Nguyen, Tuc Bailey, James Le, Trung |
| contents | Muon orthogonalizes the momentum buffer before each update, replacing its singular values with ones via Newton-Schulz iterations. This simple change lets Muon tolerate far larger learning rates and converge faster than other optimizers, but why? We show that the mechanism is spectral flattening, and develop two results around it. First, we prove that Muon's maximal stable step size scales with the average singular value of the gradient rather than the largest, which bottlenecks standard gradient descent. Second, we recast Muon as a preconditioned gradient method and show, under a Kronecker-factored curvature model, that it improves the effective convergence factor, with the improvement controlled by the spectrum of the gradient covariance. Extensive experiments validate both results: Muon remains stable at learning rates that cause SGD to diverge within the first few iterations, and reaches accuracy milestones several epochs earlier even at identical step sizes. Taken together, our results offer a principled, geometric explanation for Muon's empirical success. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_13079 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Spectral Flattening Is All Muon Needs: How Orthogonalization Controls Learning Rate and Convergence Nguyen, Tien-Phat Nguyen, Truong Truong, Minh-Phuc Nguyen, Tuc Bailey, James Le, Trung Machine Learning Artificial Intelligence Muon orthogonalizes the momentum buffer before each update, replacing its singular values with ones via Newton-Schulz iterations. This simple change lets Muon tolerate far larger learning rates and converge faster than other optimizers, but why? We show that the mechanism is spectral flattening, and develop two results around it. First, we prove that Muon's maximal stable step size scales with the average singular value of the gradient rather than the largest, which bottlenecks standard gradient descent. Second, we recast Muon as a preconditioned gradient method and show, under a Kronecker-factored curvature model, that it improves the effective convergence factor, with the improvement controlled by the spectrum of the gradient covariance. Extensive experiments validate both results: Muon remains stable at learning rates that cause SGD to diverge within the first few iterations, and reaches accuracy milestones several epochs earlier even at identical step sizes. Taken together, our results offer a principled, geometric explanation for Muon's empirical success. |
| title | Spectral Flattening Is All Muon Needs: How Orthogonalization Controls Learning Rate and Convergence |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2605.13079 |