First and Second Order Approximations to Stochastic Gradient Descent Methods with Momentum Terms
Fuente:
arXiv
Enregistré dans:
| Auteur principal: | |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866915250148737024 |
|---|---|
| author | Lu, Eric |
| author_facet | Lu, Eric |
| contents | Stochastic Gradient Descent (SGD) methods see many uses in optimization problems. Modifications to the algorithm, such as momentum-based SGD methods have been known to produce better results in certain cases. Much of this, however, is due to empirical information rather than rigorous proof. While the dynamics of gradient descent methods can be studied through continuous approximations, existing works only cover scenarios with constant learning rates or SGD without momentum terms. We present approximation results under weak assumptions for SGD that allow learning rates and momentum parameters to vary with respect to time. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_13992 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | First and Second Order Approximations to Stochastic Gradient Descent Methods with Momentum Terms Lu, Eric Machine Learning Optimization and Control Stochastic Gradient Descent (SGD) methods see many uses in optimization problems. Modifications to the algorithm, such as momentum-based SGD methods have been known to produce better results in certain cases. Much of this, however, is due to empirical information rather than rigorous proof. While the dynamics of gradient descent methods can be studied through continuous approximations, existing works only cover scenarios with constant learning rates or SGD without momentum terms. We present approximation results under weak assumptions for SGD that allow learning rates and momentum parameters to vary with respect to time. |
| title | First and Second Order Approximations to Stochastic Gradient Descent Methods with Momentum Terms |
| topic | Machine Learning Optimization and Control |
| url | https://arxiv.org/abs/2504.13992 |