Scaffold with Stochastic Gradients: New Analysis with Linear Speed-Up
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866915189901754368 |
|---|---|
| author | Mangold, Paul Durmus, Alain Dieuleveut, Aymeric Moulines, Eric |
| author_facet | Mangold, Paul Durmus, Alain Dieuleveut, Aymeric Moulines, Eric |
| contents | This paper proposes a novel analysis for the Scaffold algorithm, a popular method for dealing with data heterogeneity in federated learning. While its convergence in deterministic settings--where local control variates mitigate client drift--is well established, the impact of stochastic gradient updates on its performance is less understood. To address this problem, we first show that its global parameters and control variates define a Markov chain that converges to a stationary distribution in the Wasserstein distance. Leveraging this result, we prove that Scaffold achieves linear speed-up in the number of clients up to higher-order terms in the step size. Nevertheless, our analysis reveals that Scaffold retains a higher-order bias, similar to FedAvg, that does not decrease as the number of clients increases. This highlights opportunities for developing improved stochastic federated learning algorithms |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_07594 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Scaffold with Stochastic Gradients: New Analysis with Linear Speed-Up Mangold, Paul Durmus, Alain Dieuleveut, Aymeric Moulines, Eric Machine Learning Optimization and Control This paper proposes a novel analysis for the Scaffold algorithm, a popular method for dealing with data heterogeneity in federated learning. While its convergence in deterministic settings--where local control variates mitigate client drift--is well established, the impact of stochastic gradient updates on its performance is less understood. To address this problem, we first show that its global parameters and control variates define a Markov chain that converges to a stationary distribution in the Wasserstein distance. Leveraging this result, we prove that Scaffold achieves linear speed-up in the number of clients up to higher-order terms in the step size. Nevertheless, our analysis reveals that Scaffold retains a higher-order bias, similar to FedAvg, that does not decrease as the number of clients increases. This highlights opportunities for developing improved stochastic federated learning algorithms |
| title | Scaffold with Stochastic Gradients: New Analysis with Linear Speed-Up |
| topic | Machine Learning Optimization and Control |
| url | https://arxiv.org/abs/2503.07594 |