Constrained Stochastic Spectral Preconditioning Converges for Nonconvex Objectives
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2026
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866911673711853568 |
|---|---|
| author | Oikonomidis, Konstantinos Quan, Jan Antonakopoulos, Kimon Silveti-Falls, Antonio Cevher, Volkan Patrinos, Panagiotis |
| author_facet | Oikonomidis, Konstantinos Quan, Jan Antonakopoulos, Kimon Silveti-Falls, Antonio Cevher, Volkan Patrinos, Panagiotis |
| contents | In this work, we develop proximal preconditioned gradient methods with a focus on spectral gradient methods providing a proximal extension to the Muon and Scion optimizers. We introduce a family of stochastic algorithms that can handle a wide variety of convex and nonconvex constraints and study its convergence under heavy-tailed noise, through a novel analysis tailored to the geometry of the proposed methods. We further propose a variance-reduced version, which achieves faster convergence under standard noise assumptions. Finally, we show that the polynomial iterations used in Muon are more accurately captured by a nonlinear preconditioner than by the ideal matrix sign, leading to a convergence analysis that more faithfully reflects practical implementations. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_11850 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Constrained Stochastic Spectral Preconditioning Converges for Nonconvex Objectives Oikonomidis, Konstantinos Quan, Jan Antonakopoulos, Kimon Silveti-Falls, Antonio Cevher, Volkan Patrinos, Panagiotis Optimization and Control Machine Learning In this work, we develop proximal preconditioned gradient methods with a focus on spectral gradient methods providing a proximal extension to the Muon and Scion optimizers. We introduce a family of stochastic algorithms that can handle a wide variety of convex and nonconvex constraints and study its convergence under heavy-tailed noise, through a novel analysis tailored to the geometry of the proposed methods. We further propose a variance-reduced version, which achieves faster convergence under standard noise assumptions. Finally, we show that the polynomial iterations used in Muon are more accurately captured by a nonlinear preconditioner than by the ideal matrix sign, leading to a convergence analysis that more faithfully reflects practical implementations. |
| title | Constrained Stochastic Spectral Preconditioning Converges for Nonconvex Objectives |
| topic | Optimization and Control Machine Learning |
| url | https://arxiv.org/abs/2605.11850 |