Byzantine Robustness and Partial Participation Can Be Achieved at Once: Just Clip Gradient Differences
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913380895293440 |
|---|---|
| author | Malinovsky, Grigory Richtárik, Peter Horváth, Samuel Gorbunov, Eduard |
| author_facet | Malinovsky, Grigory Richtárik, Peter Horváth, Samuel Gorbunov, Eduard |
| contents | Distributed learning has emerged as a leading paradigm for training large machine learning models. However, in real-world scenarios, participants may be unreliable or malicious, posing a significant challenge to the integrity and accuracy of the trained models. Byzantine fault tolerance mechanisms have been proposed to address these issues, but they often assume full participation from all clients, which is not always practical due to the unavailability of some clients or communication constraints. In our work, we propose the first distributed method with client sampling and provable tolerance to Byzantine workers. The key idea behind the developed method is the use of gradient clipping to control stochastic gradient differences in recursive variance reduction. This allows us to bound the potential harm caused by Byzantine workers, even during iterations when all sampled clients are Byzantine. Furthermore, we incorporate communication compression into the method to enhance communication efficiency. Under general assumptions, we prove convergence rates for the proposed method that match the existing state-of-the-art (SOTA) theoretical results. We also propose a heuristic on adjusting any Byzantine-robust method to a partial participation scenario via clipping. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2311_14127 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Byzantine Robustness and Partial Participation Can Be Achieved at Once: Just Clip Gradient Differences Malinovsky, Grigory Richtárik, Peter Horváth, Samuel Gorbunov, Eduard Machine Learning Artificial Intelligence Distributed, Parallel, and Cluster Computing Optimization and Control Distributed learning has emerged as a leading paradigm for training large machine learning models. However, in real-world scenarios, participants may be unreliable or malicious, posing a significant challenge to the integrity and accuracy of the trained models. Byzantine fault tolerance mechanisms have been proposed to address these issues, but they often assume full participation from all clients, which is not always practical due to the unavailability of some clients or communication constraints. In our work, we propose the first distributed method with client sampling and provable tolerance to Byzantine workers. The key idea behind the developed method is the use of gradient clipping to control stochastic gradient differences in recursive variance reduction. This allows us to bound the potential harm caused by Byzantine workers, even during iterations when all sampled clients are Byzantine. Furthermore, we incorporate communication compression into the method to enhance communication efficiency. Under general assumptions, we prove convergence rates for the proposed method that match the existing state-of-the-art (SOTA) theoretical results. We also propose a heuristic on adjusting any Byzantine-robust method to a partial participation scenario via clipping. |
| title | Byzantine Robustness and Partial Participation Can Be Achieved at Once: Just Clip Gradient Differences |
| topic | Machine Learning Artificial Intelligence Distributed, Parallel, and Cluster Computing Optimization and Control |
| url | https://arxiv.org/abs/2311.14127 |