SLowcal-SGD: Slow Query Points Improve Local-SGD for Stochastic Convex Optimization
Fuente:
arXiv
Salvato in:
| Autori principali: | , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2023
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866912192711884800 |
|---|---|
| author | Dahan, Tehila Levy, Kfir Y. |
| author_facet | Dahan, Tehila Levy, Kfir Y. |
| contents | We consider distributed learning scenarios where M machines interact with a parameter server along several communication rounds in order to minimize a joint objective function. Focusing on the heterogeneous case, where different machines may draw samples from different data-distributions, we design the first local update method that provably benefits over the two most prominent distributed baselines: namely Minibatch-SGD and Local-SGD. Key to our approach is a slow querying technique that we customize to the distributed setting, which in turn enables a better mitigation of the bias caused by local updates. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2304_04169 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | SLowcal-SGD: Slow Query Points Improve Local-SGD for Stochastic Convex Optimization Dahan, Tehila Levy, Kfir Y. Machine Learning Optimization and Control We consider distributed learning scenarios where M machines interact with a parameter server along several communication rounds in order to minimize a joint objective function. Focusing on the heterogeneous case, where different machines may draw samples from different data-distributions, we design the first local update method that provably benefits over the two most prominent distributed baselines: namely Minibatch-SGD and Local-SGD. Key to our approach is a slow querying technique that we customize to the distributed setting, which in turn enables a better mitigation of the bias caused by local updates. |
| title | SLowcal-SGD: Slow Query Points Improve Local-SGD for Stochastic Convex Optimization |
| topic | Machine Learning Optimization and Control |
| url | https://arxiv.org/abs/2304.04169 |