Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2505.10889 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915289789104128 |
|---|---|
| author | Cheng, Difei Jin, Ruinan Qiao, Hong Zhang, Bo |
| author_facet | Cheng, Difei Jin, Ruinan Qiao, Hong Zhang, Bo |
| contents | Distributed stochastic gradient methods are widely used to preserve data privacy and ensure scalability in large-scale learning tasks. While existing theory on distributed momentum Stochastic Gradient Descent (mSGD) mainly focuses on time-averaged convergence, the more practical last-iterate convergence remains underexplored. In this work, we analyze the last-iterate convergence behavior of distributed mSGD in non-convex settings under the classical Robbins-Monro step-size schedule. We prove both almost sure convergence and $L_2$ convergence of the last iterate, and derive convergence rates. We further show that momentum can accelerate early-stage convergence, and provide experiments to support our theory. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_10889 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Convergence Analysis of the Last Iterate in Distributed Stochastic Gradient Descent with Momentum Cheng, Difei Jin, Ruinan Qiao, Hong Zhang, Bo Optimization and Control 40G15 G.1.0 Distributed stochastic gradient methods are widely used to preserve data privacy and ensure scalability in large-scale learning tasks. While existing theory on distributed momentum Stochastic Gradient Descent (mSGD) mainly focuses on time-averaged convergence, the more practical last-iterate convergence remains underexplored. In this work, we analyze the last-iterate convergence behavior of distributed mSGD in non-convex settings under the classical Robbins-Monro step-size schedule. We prove both almost sure convergence and $L_2$ convergence of the last iterate, and derive convergence rates. We further show that momentum can accelerate early-stage convergence, and provide experiments to support our theory. |
| title | Convergence Analysis of the Last Iterate in Distributed Stochastic Gradient Descent with Momentum |
| topic | Optimization and Control 40G15 G.1.0 |
| url | https://arxiv.org/abs/2505.10889 |