Local MixVR: Breaking the Communication-Sample Dependence in Distributed Learning
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2026
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866917551310635008 |
|---|---|
| author | Dahan, Tehila Hamoud, Bassel Reshef, Roie Jaggi, Martin Levy, Kfir Y. |
| author_facet | Dahan, Tehila Hamoud, Bassel Reshef, Roie Jaggi, Martin Levy, Kfir Y. |
| contents | Communication overhead is a crucial bottleneck in scalable distributed learning. While existing methods aim to efficiently utilize data points, such as Local SGD, Minibatch SGD, and their accelerated variants, they still exhibit communication-round complexity that scales with the total number of samples $N$. In this paper, we introduce Local MixVR, a distributed framework that integrates local updates with variance-reduction techniques to mitigate local noise. We show that Local MixVR is the first distributed method to eliminate the dependence of communication complexity on $N$, achieving a complexity that scales only with the number of workers $M$. In common regimes where $M<O\left(N^{1/4}\right)$, Local MixVR outperforms the state-of-the-art Minibatch Accelerated SGD baseline, bridging a long-standing gap in distributed optimization and establishing a new paradigm for communication-efficient training. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2606_01128 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Local MixVR: Breaking the Communication-Sample Dependence in Distributed Learning Dahan, Tehila Hamoud, Bassel Reshef, Roie Jaggi, Martin Levy, Kfir Y. Machine Learning Communication overhead is a crucial bottleneck in scalable distributed learning. While existing methods aim to efficiently utilize data points, such as Local SGD, Minibatch SGD, and their accelerated variants, they still exhibit communication-round complexity that scales with the total number of samples $N$. In this paper, we introduce Local MixVR, a distributed framework that integrates local updates with variance-reduction techniques to mitigate local noise. We show that Local MixVR is the first distributed method to eliminate the dependence of communication complexity on $N$, achieving a complexity that scales only with the number of workers $M$. In common regimes where $M<O\left(N^{1/4}\right)$, Local MixVR outperforms the state-of-the-art Minibatch Accelerated SGD baseline, bridging a long-standing gap in distributed optimization and establishing a new paradigm for communication-efficient training. |
| title | Local MixVR: Breaking the Communication-Sample Dependence in Distributed Learning |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2606.01128 |