Exponential convergence rates for momentum stochastic gradient descent in the overparametrized setting
Fuente:
arXiv
Guardado en:
| Autores principales: | , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2023
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866913572187013120 |
|---|---|
| author | Gess, Benjamin Kassing, Sebastian |
| author_facet | Gess, Benjamin Kassing, Sebastian |
| contents | We prove explicit bounds on the exponential rate of convergence for the momentum stochastic gradient descent scheme (MSGD) for arbitrary, fixed hyperparameters (learning rate, friction parameter) and its continuous-in-time counterpart in the context of non-convex optimization. In the small step-size regime and in the case of flat minima or large noise intensities, these bounds prove faster convergence of MSGD compared to plain stochastic gradient descent (SGD). The results are shown for objective functions satisfying a local Polyak-Lojasiewicz inequality and under assumptions on the variance of MSGD that are satisfied in overparametrized settings. Moreover, we analyze the optimal choice of the friction parameter and show that the MSGD process almost surely converges to a local minimum. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2302_03550 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Exponential convergence rates for momentum stochastic gradient descent in the overparametrized setting Gess, Benjamin Kassing, Sebastian Optimization and Control Machine Learning Probability Primary 90C15, Secondary 68T07, 90C26, 62L20 We prove explicit bounds on the exponential rate of convergence for the momentum stochastic gradient descent scheme (MSGD) for arbitrary, fixed hyperparameters (learning rate, friction parameter) and its continuous-in-time counterpart in the context of non-convex optimization. In the small step-size regime and in the case of flat minima or large noise intensities, these bounds prove faster convergence of MSGD compared to plain stochastic gradient descent (SGD). The results are shown for objective functions satisfying a local Polyak-Lojasiewicz inequality and under assumptions on the variance of MSGD that are satisfied in overparametrized settings. Moreover, we analyze the optimal choice of the friction parameter and show that the MSGD process almost surely converges to a local minimum. |
| title | Exponential convergence rates for momentum stochastic gradient descent in the overparametrized setting |
| topic | Optimization and Control Machine Learning Probability Primary 90C15, Secondary 68T07, 90C26, 62L20 |
| url | https://arxiv.org/abs/2302.03550 |