Gradient correlation is a key ingredient to accelerate SGD with momentum
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866909464978784256 |
|---|---|
| author | Hermant, Julien Renaud, Marien Aujol, Jean-François Dossal, Charles Rondepierre, Aude |
| author_facet | Hermant, Julien Renaud, Marien Aujol, Jean-François Dossal, Charles Rondepierre, Aude |
| contents | Empirically, it has been observed that adding momentum to Stochastic Gradient Descent (SGD) accelerates the convergence of the algorithm. However, the literature has been rather pessimistic, even in the case of convex functions, about the possibility of theoretically proving this observation. We investigate the possibility of obtaining accelerated convergence of the Stochastic Nesterov Accelerated Gradient (SNAG), a momentum-based version of SGD, when minimizing a sum of functions in a convex setting. We demonstrate that the average correlation between gradients allows to verify the strong growth condition, which is the key ingredient to obtain acceleration with SNAG. Numerical experiments, both in linear regression and deep neural network optimization, confirm in practice our theoretical results. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_07870 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Gradient correlation is a key ingredient to accelerate SGD with momentum Hermant, Julien Renaud, Marien Aujol, Jean-François Dossal, Charles Rondepierre, Aude Optimization and Control Empirically, it has been observed that adding momentum to Stochastic Gradient Descent (SGD) accelerates the convergence of the algorithm. However, the literature has been rather pessimistic, even in the case of convex functions, about the possibility of theoretically proving this observation. We investigate the possibility of obtaining accelerated convergence of the Stochastic Nesterov Accelerated Gradient (SNAG), a momentum-based version of SGD, when minimizing a sum of functions in a convex setting. We demonstrate that the average correlation between gradients allows to verify the strong growth condition, which is the key ingredient to obtain acceleration with SNAG. Numerical experiments, both in linear regression and deep neural network optimization, confirm in practice our theoretical results. |
| title | Gradient correlation is a key ingredient to accelerate SGD with momentum |
| topic | Optimization and Control |
| url | https://arxiv.org/abs/2410.07870 |