Effective continuous equations for adaptive SGD: a stochastic analysis view
Fuente:
arXiv
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918148271243264 |
|---|---|
| author | Callisti, Luca Romito, Marco Triggiano, Francesco |
| author_facet | Callisti, Luca Romito, Marco Triggiano, Francesco |
| contents | We present a theoretical analysis of some popular adaptive Stochastic Gradient Descent (SGD) methods in the small learning rate regime. Using the stochastic modified equations framework introduced by Li et al., we derive effective continuous stochastic dynamics for these methods. Our key contribution is that sampling-induced noise in SGD manifests in the limit as independent Brownian motions driving the parameter and gradient second momentum evolutions. Furthermore, extending the approach of Malladi et al., we investigate scaling rules between the learning rate and key hyperparameters in adaptive methods, characterising all non-trivial limiting dynamics. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_21614 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Effective continuous equations for adaptive SGD: a stochastic analysis view Callisti, Luca Romito, Marco Triggiano, Francesco Machine Learning Probability We present a theoretical analysis of some popular adaptive Stochastic Gradient Descent (SGD) methods in the small learning rate regime. Using the stochastic modified equations framework introduced by Li et al., we derive effective continuous stochastic dynamics for these methods. Our key contribution is that sampling-induced noise in SGD manifests in the limit as independent Brownian motions driving the parameter and gradient second momentum evolutions. Furthermore, extending the approach of Malladi et al., we investigate scaling rules between the learning rate and key hyperparameters in adaptive methods, characterising all non-trivial limiting dynamics. |
| title | Effective continuous equations for adaptive SGD: a stochastic analysis view |
| topic | Machine Learning Probability |
| url | https://arxiv.org/abs/2509.21614 |