Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Udgivet: |
2022
|
| Fag: | |
| Online adgang: | https://arxiv.org/abs/2210.01708 |
| Tags: |
Tilføj Tag
Ingen Tags, Vær først til at tagge denne postø!
|
| _version_ | 1866909439549767680 |
|---|---|
| author | Sun, Guangyu Khalid, Umar Mendieta, Matias Wang, Pu Chen, Chen |
| author_facet | Sun, Guangyu Khalid, Umar Mendieta, Matias Wang, Pu Chen, Chen |
| contents | Federated learning (FL) has emerged as a promising paradigm for enabling the collaborative training of models without centralized access to the raw data on local devices. In the typical FL paradigm (e.g., FedAvg), model weights are sent to and from the server each round to participating clients. Recently, the use of small pre-trained models has been shown to be effective in federated learning optimization and improving convergence. However, recent state-of-the-art pre-trained models are getting more capable but also have more parameters, known as the "Foundation Models." In conventional FL, sharing the enormous model weights can quickly put a massive communication burden on the system, especially if more capable models are employed. Can we find a solution to enable those strong and readily available pre-trained models in FL to achieve excellent performance while simultaneously reducing the communication burden? To this end, we investigate the use of parameter-efficient fine-tuning in federated learning and thus introduce a new framework: FedPEFT. Specifically, we systemically evaluate the performance of FedPEFT across a variety of client stability, data distribution, and differential privacy settings. By only locally tuning and globally sharing a small portion of the model weights, significant reductions in the total communication overhead can be achieved while maintaining competitive or even better performance in a wide range of federated learning scenarios, providing insight into a new paradigm for practical and effective federated systems. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2210_01708 |
| institution | arXiv |
| publishDate | 2022 |
| record_format | arxiv |
| spellingShingle | Exploring Parameter-Efficient Fine-Tuning to Enable Foundation Models in Federated Learning Sun, Guangyu Khalid, Umar Mendieta, Matias Wang, Pu Chen, Chen Machine Learning Computer Vision and Pattern Recognition Federated learning (FL) has emerged as a promising paradigm for enabling the collaborative training of models without centralized access to the raw data on local devices. In the typical FL paradigm (e.g., FedAvg), model weights are sent to and from the server each round to participating clients. Recently, the use of small pre-trained models has been shown to be effective in federated learning optimization and improving convergence. However, recent state-of-the-art pre-trained models are getting more capable but also have more parameters, known as the "Foundation Models." In conventional FL, sharing the enormous model weights can quickly put a massive communication burden on the system, especially if more capable models are employed. Can we find a solution to enable those strong and readily available pre-trained models in FL to achieve excellent performance while simultaneously reducing the communication burden? To this end, we investigate the use of parameter-efficient fine-tuning in federated learning and thus introduce a new framework: FedPEFT. Specifically, we systemically evaluate the performance of FedPEFT across a variety of client stability, data distribution, and differential privacy settings. By only locally tuning and globally sharing a small portion of the model weights, significant reductions in the total communication overhead can be achieved while maintaining competitive or even better performance in a wide range of federated learning scenarios, providing insight into a new paradigm for practical and effective federated systems. |
| title | Exploring Parameter-Efficient Fine-Tuning to Enable Foundation Models in Federated Learning |
| topic | Machine Learning Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2210.01708 |