Double Momentum and Error Feedback for Clipping with Fast Rates and Differential Privacy
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866910041850773504 |
|---|---|
| author | Islamov, Rustem Horvath, Samuel Lucchi, Aurelien Richtarik, Peter Gorbunov, Eduard |
| author_facet | Islamov, Rustem Horvath, Samuel Lucchi, Aurelien Richtarik, Peter Gorbunov, Eduard |
| contents | Strong Differential Privacy (DP) and Optimization guarantees are two desirable properties for a method in Federated Learning (FL). However, existing algorithms do not achieve both properties at once: they either have optimal DP guarantees but rely on restrictive assumptions such as bounded gradients/bounded data heterogeneity, or they ensure strong optimization performance but lack DP guarantees. To address this gap in the literature, we propose and analyze a new method called Clip21-SGD2M based on a novel combination of clipping, heavy-ball momentum, and Error Feedback. In particular, for non-convex smooth distributed problems with clients having arbitrarily heterogeneous data, we prove that Clip21-SGD2M has optimal convergence rate and also near optimal (local-)DP neighborhood. Our numerical experiments on non-convex logistic regression and training of neural networks highlight the superiority of Clip21-SGD2M over baselines in terms of the optimization performance for a given DP-budget. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_11682 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Double Momentum and Error Feedback for Clipping with Fast Rates and Differential Privacy Islamov, Rustem Horvath, Samuel Lucchi, Aurelien Richtarik, Peter Gorbunov, Eduard Machine Learning Optimization and Control Strong Differential Privacy (DP) and Optimization guarantees are two desirable properties for a method in Federated Learning (FL). However, existing algorithms do not achieve both properties at once: they either have optimal DP guarantees but rely on restrictive assumptions such as bounded gradients/bounded data heterogeneity, or they ensure strong optimization performance but lack DP guarantees. To address this gap in the literature, we propose and analyze a new method called Clip21-SGD2M based on a novel combination of clipping, heavy-ball momentum, and Error Feedback. In particular, for non-convex smooth distributed problems with clients having arbitrarily heterogeneous data, we prove that Clip21-SGD2M has optimal convergence rate and also near optimal (local-)DP neighborhood. Our numerical experiments on non-convex logistic regression and training of neural networks highlight the superiority of Clip21-SGD2M over baselines in terms of the optimization performance for a given DP-budget. |
| title | Double Momentum and Error Feedback for Clipping with Fast Rates and Differential Privacy |
| topic | Machine Learning Optimization and Control |
| url | https://arxiv.org/abs/2502.11682 |