Step-Size Stability in Stochastic Optimization: A Theoretical Perspective
Fuente:
arXiv
Salvato in:
| Autori principali: | , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2026
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866913163566383104 |
|---|---|
| author | Schaipp, Fabian Gower, Robert M. Taylor, Adrien |
| author_facet | Schaipp, Fabian Gower, Robert M. Taylor, Adrien |
| contents | We present a theoretical analysis of stochastic optimization methods in terms of their sensitivity with respect to the step size. We identify a key quantity that, for each method, describes how the performance degrades as the step size becomes too large. For convex problems, we show that this quantity directly impacts the suboptimality bound of the method. Most importantly, our analysis provides direct theoretical evidence that adaptive step-size methods, such as SPS or NGN, are more robust than SGD. This allows us to quantify the advantage of these adaptive methods beyond empirical evaluation. Finally, we show through experiments that our theoretical bound qualitatively mirrors the actual performance as a function of the step size, even for non-convex problems. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_09842 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Step-Size Stability in Stochastic Optimization: A Theoretical Perspective Schaipp, Fabian Gower, Robert M. Taylor, Adrien Optimization and Control Machine Learning 90C26 We present a theoretical analysis of stochastic optimization methods in terms of their sensitivity with respect to the step size. We identify a key quantity that, for each method, describes how the performance degrades as the step size becomes too large. For convex problems, we show that this quantity directly impacts the suboptimality bound of the method. Most importantly, our analysis provides direct theoretical evidence that adaptive step-size methods, such as SPS or NGN, are more robust than SGD. This allows us to quantify the advantage of these adaptive methods beyond empirical evaluation. Finally, we show through experiments that our theoretical bound qualitatively mirrors the actual performance as a function of the step size, even for non-convex problems. |
| title | Step-Size Stability in Stochastic Optimization: A Theoretical Perspective |
| topic | Optimization and Control Machine Learning 90C26 |
| url | https://arxiv.org/abs/2602.09842 |