Federated Learning on Non-iid Data via Local and Global Distillation
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910309213536256 |
|---|---|
| author | Zheng, Xiaolin Ying, Senci Zheng, Fei Yin, Jianwei Zheng, Longfei Chen, Chaochao Dong, Fengqin |
| author_facet | Zheng, Xiaolin Ying, Senci Zheng, Fei Yin, Jianwei Zheng, Longfei Chen, Chaochao Dong, Fengqin |
| contents | Most existing federated learning algorithms are based on the vanilla FedAvg scheme. However, with the increase of data complexity and the number of model parameters, the amount of communication traffic and the number of iteration rounds for training such algorithms increases significantly, especially in non-independently and homogeneously distributed scenarios, where they do not achieve satisfactory performance. In this work, we propose FedND: federated learning with noise distillation. The main idea is to use knowledge distillation to optimize the model training process. In the client, we propose a self-distillation method to train the local model. In the server, we generate noisy samples for each client and use them to distill other clients. Finally, the global model is obtained by the aggregation of local models. Experimental results show that the algorithm achieves the best performance and is more communication-efficient than state-of-the-art methods. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2306_14443 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Federated Learning on Non-iid Data via Local and Global Distillation Zheng, Xiaolin Ying, Senci Zheng, Fei Yin, Jianwei Zheng, Longfei Chen, Chaochao Dong, Fengqin Machine Learning Cryptography and Security Most existing federated learning algorithms are based on the vanilla FedAvg scheme. However, with the increase of data complexity and the number of model parameters, the amount of communication traffic and the number of iteration rounds for training such algorithms increases significantly, especially in non-independently and homogeneously distributed scenarios, where they do not achieve satisfactory performance. In this work, we propose FedND: federated learning with noise distillation. The main idea is to use knowledge distillation to optimize the model training process. In the client, we propose a self-distillation method to train the local model. In the server, we generate noisy samples for each client and use them to distill other clients. Finally, the global model is obtained by the aggregation of local models. Experimental results show that the algorithm achieves the best performance and is more communication-efficient than state-of-the-art methods. |
| title | Federated Learning on Non-iid Data via Local and Global Distillation |
| topic | Machine Learning Cryptography and Security |
| url | https://arxiv.org/abs/2306.14443 |