Minibatch and Local SGD: Algorithmic Stability and Linear Speedup in Generalization
Fuente:
arXiv
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915544816418816 |
|---|---|
| author | Lei, Yunwen Sun, Tao Liu, Mingrui |
| author_facet | Lei, Yunwen Sun, Tao Liu, Mingrui |
| contents | The increasing scale of data propels the popularity of leveraging parallelism to speed up the optimization. Minibatch stochastic gradient descent (minibatch SGD) and local SGD are two popular methods for parallel optimization. The existing theoretical studies show a linear speedup of these methods with respect to the number of machines, which, however, is measured by optimization errors in a multi-pass setting. As a comparison, the stability and generalization of these methods are much less studied. In this paper, we study the stability and generalization analysis of minibatch and local SGD to understand their learnability by introducing an expectation-variance decomposition. We incorporate training errors into the stability analysis, which shows how small training errors help generalization for overparameterized models. We show minibatch and local SGD achieve a linear speedup to attain the optimal risk bounds. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2310_01139 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Minibatch and Local SGD: Algorithmic Stability and Linear Speedup in Generalization Lei, Yunwen Sun, Tao Liu, Mingrui Machine Learning Artificial Intelligence The increasing scale of data propels the popularity of leveraging parallelism to speed up the optimization. Minibatch stochastic gradient descent (minibatch SGD) and local SGD are two popular methods for parallel optimization. The existing theoretical studies show a linear speedup of these methods with respect to the number of machines, which, however, is measured by optimization errors in a multi-pass setting. As a comparison, the stability and generalization of these methods are much less studied. In this paper, we study the stability and generalization analysis of minibatch and local SGD to understand their learnability by introducing an expectation-variance decomposition. We incorporate training errors into the stability analysis, which shows how small training errors help generalization for overparameterized models. We show minibatch and local SGD achieve a linear speedup to attain the optimal risk bounds. |
| title | Minibatch and Local SGD: Algorithmic Stability and Linear Speedup in Generalization |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2310.01139 |