High Dimensional Distributed Gradient Descent with Arbitrary Number of Byzantine Attackers
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2023
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866918207168708608 |
|---|---|
| author | Liu, Wenyu Huang, Tianqiang Zhang, Pengfei Ke, Zong Min, Minghui Zhao, Puning |
| author_facet | Liu, Wenyu Huang, Tianqiang Zhang, Pengfei Ke, Zong Min, Minghui Zhao, Puning |
| contents | Adversarial attacks pose a major challenge to distributed learning systems, prompting the development of numerous robust learning methods. However, most existing approaches suffer from the curse of dimensionality, i.e. the error increases with the number of model parameters. In this paper, we make a progress towards high dimensional problems, under arbitrary number of Byzantine attackers. The cornerstone of our design is a direct high dimensional semi-verified mean estimation method. The idea is to identify a subspace with large variance. The components of the mean value perpendicular to this subspace are estimated using corrupted gradient vectors uploaded from worker machines, while the components within this subspace are estimated using auxiliary dataset. As a result, a combination of large corrupted dataset and small clean dataset yields significantly better performance than using them separately. We then apply this method as the aggregator for distributed learning problems. The theoretical analysis shows that compared with existing solutions, our method gets rid of $\sqrt{d}$ dependence on the dimensionality, and achieves minimax optimal statistical rates. Numerical results validate our theory as well as the effectiveness of the proposed method. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2307_13352 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | High Dimensional Distributed Gradient Descent with Arbitrary Number of Byzantine Attackers Liu, Wenyu Huang, Tianqiang Zhang, Pengfei Ke, Zong Min, Minghui Zhao, Puning Machine Learning Cryptography and Security Distributed, Parallel, and Cluster Computing Adversarial attacks pose a major challenge to distributed learning systems, prompting the development of numerous robust learning methods. However, most existing approaches suffer from the curse of dimensionality, i.e. the error increases with the number of model parameters. In this paper, we make a progress towards high dimensional problems, under arbitrary number of Byzantine attackers. The cornerstone of our design is a direct high dimensional semi-verified mean estimation method. The idea is to identify a subspace with large variance. The components of the mean value perpendicular to this subspace are estimated using corrupted gradient vectors uploaded from worker machines, while the components within this subspace are estimated using auxiliary dataset. As a result, a combination of large corrupted dataset and small clean dataset yields significantly better performance than using them separately. We then apply this method as the aggregator for distributed learning problems. The theoretical analysis shows that compared with existing solutions, our method gets rid of $\sqrt{d}$ dependence on the dimensionality, and achieves minimax optimal statistical rates. Numerical results validate our theory as well as the effectiveness of the proposed method. |
| title | High Dimensional Distributed Gradient Descent with Arbitrary Number of Byzantine Attackers |
| topic | Machine Learning Cryptography and Security Distributed, Parallel, and Cluster Computing |
| url | https://arxiv.org/abs/2307.13352 |