Random Scaling and Momentum for Non-smooth Non-convex Optimization
Fuente:
arXiv
Saved in:
| Main Authors: | , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910052059709440 |
|---|---|
| author | Zhang, Qinzi Cutkosky, Ashok |
| author_facet | Zhang, Qinzi Cutkosky, Ashok |
| contents | Training neural networks requires optimizing a loss function that may be highly irregular, and in particular neither convex nor smooth. Popular training algorithms are based on stochastic gradient descent with momentum (SGDM), for which classical analysis applies only if the loss is either convex or smooth. We show that a very small modification to SGDM closes this gap: simply scale the update at each time point by an exponentially distributed random scalar. The resulting algorithm achieves optimal convergence guarantees. Intriguingly, this result is not derived by a specific analysis of SGDM: instead, it falls naturally out of a more general framework for converting online convex optimization algorithms to non-convex optimization algorithms. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_09742 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Random Scaling and Momentum for Non-smooth Non-convex Optimization Zhang, Qinzi Cutkosky, Ashok Machine Learning Optimization and Control Training neural networks requires optimizing a loss function that may be highly irregular, and in particular neither convex nor smooth. Popular training algorithms are based on stochastic gradient descent with momentum (SGDM), for which classical analysis applies only if the loss is either convex or smooth. We show that a very small modification to SGDM closes this gap: simply scale the update at each time point by an exponentially distributed random scalar. The resulting algorithm achieves optimal convergence guarantees. Intriguingly, this result is not derived by a specific analysis of SGDM: instead, it falls naturally out of a more general framework for converting online convex optimization algorithms to non-convex optimization algorithms. |
| title | Random Scaling and Momentum for Non-smooth Non-convex Optimization |
| topic | Machine Learning Optimization and Control |
| url | https://arxiv.org/abs/2405.09742 |