FedImpro: Measuring and Improving Client Update in Federated Learning
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909136167370752 |
|---|---|
| author | Tang, Zhenheng Zhang, Yonggang Shi, Shaohuai Tian, Xinmei Liu, Tongliang Han, Bo Chu, Xiaowen |
| author_facet | Tang, Zhenheng Zhang, Yonggang Shi, Shaohuai Tian, Xinmei Liu, Tongliang Han, Bo Chu, Xiaowen |
| contents | Federated Learning (FL) models often experience client drift caused by heterogeneous data, where the distribution of data differs across clients. To address this issue, advanced research primarily focuses on manipulating the existing gradients to achieve more consistent client models. In this paper, we present an alternative perspective on client drift and aim to mitigate it by generating improved local models. First, we analyze the generalization contribution of local training and conclude that this generalization contribution is bounded by the conditional Wasserstein distance between the data distribution of different clients. Then, we propose FedImpro, to construct similar conditional distributions for local training. Specifically, FedImpro decouples the model into high-level and low-level components, and trains the high-level portion on reconstructed feature distributions. This approach enhances the generalization contribution and reduces the dissimilarity of gradients in FL. Experimental results show that FedImpro can help FL defend against data heterogeneity and enhance the generalization performance of the model. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_07011 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | FedImpro: Measuring and Improving Client Update in Federated Learning Tang, Zhenheng Zhang, Yonggang Shi, Shaohuai Tian, Xinmei Liu, Tongliang Han, Bo Chu, Xiaowen Machine Learning Artificial Intelligence Distributed, Parallel, and Cluster Computing Federated Learning (FL) models often experience client drift caused by heterogeneous data, where the distribution of data differs across clients. To address this issue, advanced research primarily focuses on manipulating the existing gradients to achieve more consistent client models. In this paper, we present an alternative perspective on client drift and aim to mitigate it by generating improved local models. First, we analyze the generalization contribution of local training and conclude that this generalization contribution is bounded by the conditional Wasserstein distance between the data distribution of different clients. Then, we propose FedImpro, to construct similar conditional distributions for local training. Specifically, FedImpro decouples the model into high-level and low-level components, and trains the high-level portion on reconstructed feature distributions. This approach enhances the generalization contribution and reduces the dissimilarity of gradients in FL. Experimental results show that FedImpro can help FL defend against data heterogeneity and enhance the generalization performance of the model. |
| title | FedImpro: Measuring and Improving Client Update in Federated Learning |
| topic | Machine Learning Artificial Intelligence Distributed, Parallel, and Cluster Computing |
| url | https://arxiv.org/abs/2402.07011 |