The Marginal Value of Momentum for Small Learning Rate SGD
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913316298817536 |
|---|---|
| author | Wang, Runzhe Malladi, Sadhika Wang, Tianhao Lyu, Kaifeng Li, Zhiyuan |
| author_facet | Wang, Runzhe Malladi, Sadhika Wang, Tianhao Lyu, Kaifeng Li, Zhiyuan |
| contents | Momentum is known to accelerate the convergence of gradient descent in strongly convex settings without stochastic gradient noise. In stochastic optimization, such as training neural networks, folklore suggests that momentum may help deep learning optimization by reducing the variance of the stochastic gradient update, but previous theoretical analyses do not find momentum to offer any provable acceleration. Theoretical results in this paper clarify the role of momentum in stochastic settings where the learning rate is small and gradient noise is the dominant source of instability, suggesting that SGD with and without momentum behave similarly in the short and long time horizons. Experiments show that momentum indeed has limited benefits for both optimization and generalization in practical training regimes where the optimal learning rate is not very large, including small- to medium-batch training from scratch on ImageNet and fine-tuning language models on downstream tasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2307_15196 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | The Marginal Value of Momentum for Small Learning Rate SGD Wang, Runzhe Malladi, Sadhika Wang, Tianhao Lyu, Kaifeng Li, Zhiyuan Machine Learning Optimization and Control Momentum is known to accelerate the convergence of gradient descent in strongly convex settings without stochastic gradient noise. In stochastic optimization, such as training neural networks, folklore suggests that momentum may help deep learning optimization by reducing the variance of the stochastic gradient update, but previous theoretical analyses do not find momentum to offer any provable acceleration. Theoretical results in this paper clarify the role of momentum in stochastic settings where the learning rate is small and gradient noise is the dominant source of instability, suggesting that SGD with and without momentum behave similarly in the short and long time horizons. Experiments show that momentum indeed has limited benefits for both optimization and generalization in practical training regimes where the optimal learning rate is not very large, including small- to medium-batch training from scratch on ImageNet and fine-tuning language models on downstream tasks. |
| title | The Marginal Value of Momentum for Small Learning Rate SGD |
| topic | Machine Learning Optimization and Control |
| url | https://arxiv.org/abs/2307.15196 |