Universal Gradient Methods for Stochastic Convex Optimization
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866916319646973952 |
|---|---|
| author | Rodomanov, Anton Kavis, Ali Wu, Yongtao Antonakopoulos, Kimon Cevher, Volkan |
| author_facet | Rodomanov, Anton Kavis, Ali Wu, Yongtao Antonakopoulos, Kimon Cevher, Volkan |
| contents | We develop universal gradient methods for Stochastic Convex Optimization (SCO). Our algorithms automatically adapt not only to the oracle's noise but also to the Hölder smoothness of the objective function without a priori knowledge of the particular setting. The key ingredient is a novel strategy for adjusting step-size coefficients in the Stochastic Gradient Method (SGD). Unlike AdaGrad, which accumulates gradient norms, our Universal Gradient Method accumulates appropriate combinations of gradient- and iterate differences. The resulting algorithm has state-of-the-art worst-case convergence rate guarantees for the entire Hölder class including, in particular, both nonsmooth functions and those with Lipschitz continuous gradient. We also present the Universal Fast Gradient Method for SCO enjoying optimal efficiency estimates. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_03210 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Universal Gradient Methods for Stochastic Convex Optimization Rodomanov, Anton Kavis, Ali Wu, Yongtao Antonakopoulos, Kimon Cevher, Volkan Optimization and Control We develop universal gradient methods for Stochastic Convex Optimization (SCO). Our algorithms automatically adapt not only to the oracle's noise but also to the Hölder smoothness of the objective function without a priori knowledge of the particular setting. The key ingredient is a novel strategy for adjusting step-size coefficients in the Stochastic Gradient Method (SGD). Unlike AdaGrad, which accumulates gradient norms, our Universal Gradient Method accumulates appropriate combinations of gradient- and iterate differences. The resulting algorithm has state-of-the-art worst-case convergence rate guarantees for the entire Hölder class including, in particular, both nonsmooth functions and those with Lipschitz continuous gradient. We also present the Universal Fast Gradient Method for SCO enjoying optimal efficiency estimates. |
| title | Universal Gradient Methods for Stochastic Convex Optimization |
| topic | Optimization and Control |
| url | https://arxiv.org/abs/2402.03210 |