Salvato in:
| Autori principali: | , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | https://arxiv.org/abs/2405.16954 |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866910763271061504 |
|---|---|
| author | Qiu, Junwen Ma, Bohao Milzarek, Andre |
| author_facet | Qiu, Junwen Ma, Bohao Milzarek, Andre |
| contents | The stochastic gradient descent method with momentum (SGDM) is a common approach for solving large-scale and stochastic optimization problems. Despite its popularity, the convergence behavior of SGDM remains less understood in nonconvex scenarios. This is primarily due to the absence of a sufficient descent property and challenges in simultaneously controlling the momentum and stochastic errors in an almost sure sense. To address these challenges, we investigate the behavior of SGDM over specific time windows, rather than examining the descent of consecutive iterates as in traditional studies. This time window-based approach simplifies the convergence analysis and enables us to establish the iterate convergence result for SGDM under the Łojasiewicz property. We further provide local convergence rates which depend on the underlying Łojasiewicz exponent and the utilized step size schemes. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_16954 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Convergence of SGD with momentum in the nonconvex case: A time window-based analysis Qiu, Junwen Ma, Bohao Milzarek, Andre Optimization and Control Machine Learning The stochastic gradient descent method with momentum (SGDM) is a common approach for solving large-scale and stochastic optimization problems. Despite its popularity, the convergence behavior of SGDM remains less understood in nonconvex scenarios. This is primarily due to the absence of a sufficient descent property and challenges in simultaneously controlling the momentum and stochastic errors in an almost sure sense. To address these challenges, we investigate the behavior of SGDM over specific time windows, rather than examining the descent of consecutive iterates as in traditional studies. This time window-based approach simplifies the convergence analysis and enables us to establish the iterate convergence result for SGDM under the Łojasiewicz property. We further provide local convergence rates which depend on the underlying Łojasiewicz exponent and the utilized step size schemes. |
| title | Convergence of SGD with momentum in the nonconvex case: A time window-based analysis |
| topic | Optimization and Control Machine Learning |
| url | https://arxiv.org/abs/2405.16954 |