Error analysis for stochastic gradient optimization schemes using modified equations
Fuente:
arXiv
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914278633635840 |
|---|---|
| author | Bréhier, Charles-Edouard Dambrine, Marc En-Nebbazi, Nassim |
| author_facet | Bréhier, Charles-Edouard Dambrine, Marc En-Nebbazi, Nassim |
| contents | We consider a class of stochastic gradient optimization schemes. Assuming that the objective function is strongly convex, we prove weak error estimates which are uniform in time for the error between the solution of the numerical scheme, and the solutions of continuous-time modified (or high-resolution) differential equations at first and second orders, with respect to the time-step size. At first order, the modified equation is deterministic, whereas at second order the modified equation is stochastic and depends on a modified objective function. We go beyond existing results where the error estimates have been considered only on finite time intervals and were not uniform in time. This allows us to then provide a rigorous complexity analysis of the method in the large time and small time-step size regimes. We provide numerical experiments to illustrate the convergence results. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_05538 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Error analysis for stochastic gradient optimization schemes using modified equations Bréhier, Charles-Edouard Dambrine, Marc En-Nebbazi, Nassim Numerical Analysis Optimization and Control Probability We consider a class of stochastic gradient optimization schemes. Assuming that the objective function is strongly convex, we prove weak error estimates which are uniform in time for the error between the solution of the numerical scheme, and the solutions of continuous-time modified (or high-resolution) differential equations at first and second orders, with respect to the time-step size. At first order, the modified equation is deterministic, whereas at second order the modified equation is stochastic and depends on a modified objective function. We go beyond existing results where the error estimates have been considered only on finite time intervals and were not uniform in time. This allows us to then provide a rigorous complexity analysis of the method in the large time and small time-step size regimes. We provide numerical experiments to illustrate the convergence results. |
| title | Error analysis for stochastic gradient optimization schemes using modified equations |
| topic | Numerical Analysis Optimization and Control Probability |
| url | https://arxiv.org/abs/2411.05538 |