ISFL: Federated Learning for Non-i.i.d. Data with Local Importance Sampling
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2022
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866909198511505408 |
|---|---|
| author | Zhu, Zheqi Shi, Yuchen Fan, Pingyi Peng, Chenghui Letaief, Khaled B. |
| author_facet | Zhu, Zheqi Shi, Yuchen Fan, Pingyi Peng, Chenghui Letaief, Khaled B. |
| contents | As a promising learning paradigm integrating computation and communication, federated learning (FL) proceeds the local training and the periodic sharing from distributed clients. Due to the non-i.i.d. data distribution on clients, FL model suffers from the gradient diversity, poor performance, bad convergence, etc. In this work, we aim to tackle this key issue by adopting importance sampling (IS) for local training. We propose importance sampling federated learning (ISFL), an explicit framework with theoretical guarantees. Firstly, we derive the convergence theorem of ISFL to involve the effects of local importance sampling. Then, we formulate the problem of selecting optimal IS weights and obtain the theoretical solutions. We also employ a water-filling method to calculate the IS weights and develop the ISFL algorithms. The experimental results on CIFAR-10 fit the proposed theorems well and verify that ISFL reaps better performance, sampling efficiency, as well as explainability on non-i.i.d. data. To the best of our knowledge, ISFL is the first non-i.i.d. FL solution from the local sampling aspect which exhibits theoretical compatibility with neural network models. Furthermore, as a local sampling approach, ISFL can be easily migrated into other emerging FL frameworks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2210_02119 |
| institution | arXiv |
| publishDate | 2022 |
| record_format | arxiv |
| spellingShingle | ISFL: Federated Learning for Non-i.i.d. Data with Local Importance Sampling Zhu, Zheqi Shi, Yuchen Fan, Pingyi Peng, Chenghui Letaief, Khaled B. Machine Learning Artificial Intelligence Distributed, Parallel, and Cluster Computing Multiagent Systems As a promising learning paradigm integrating computation and communication, federated learning (FL) proceeds the local training and the periodic sharing from distributed clients. Due to the non-i.i.d. data distribution on clients, FL model suffers from the gradient diversity, poor performance, bad convergence, etc. In this work, we aim to tackle this key issue by adopting importance sampling (IS) for local training. We propose importance sampling federated learning (ISFL), an explicit framework with theoretical guarantees. Firstly, we derive the convergence theorem of ISFL to involve the effects of local importance sampling. Then, we formulate the problem of selecting optimal IS weights and obtain the theoretical solutions. We also employ a water-filling method to calculate the IS weights and develop the ISFL algorithms. The experimental results on CIFAR-10 fit the proposed theorems well and verify that ISFL reaps better performance, sampling efficiency, as well as explainability on non-i.i.d. data. To the best of our knowledge, ISFL is the first non-i.i.d. FL solution from the local sampling aspect which exhibits theoretical compatibility with neural network models. Furthermore, as a local sampling approach, ISFL can be easily migrated into other emerging FL frameworks. |
| title | ISFL: Federated Learning for Non-i.i.d. Data with Local Importance Sampling |
| topic | Machine Learning Artificial Intelligence Distributed, Parallel, and Cluster Computing Multiagent Systems |
| url | https://arxiv.org/abs/2210.02119 |