A Survey on Efficient Federated Learning Methods for Foundation Model Training
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866914527837159424 |
|---|---|
| author | Woisetschläger, Herbert Isenko, Alexander Wang, Shiqiang Mayer, Ruben Jacobsen, Hans-Arno |
| author_facet | Woisetschläger, Herbert Isenko, Alexander Wang, Shiqiang Mayer, Ruben Jacobsen, Hans-Arno |
| contents | Federated Learning (FL) has become an established technique to facilitate privacy-preserving collaborative training across a multitude of clients. However, new approaches to FL often discuss their contributions involving small deep-learning models only and focus on training full models on clients. In the wake of Foundation Models (FM), the reality is different for many deep learning applications. Typically, FMs have already been pre-trained across a wide variety of tasks and can be fine-tuned to specific downstream tasks over significantly smaller datasets than required for full model training. However, access to such datasets is often challenging. By its design, FL can help to open data silos. With this survey, we introduce a novel taxonomy focused on computational and communication efficiency, the vital elements to make use of FMs in FL systems. We discuss the benefits and drawbacks of parameter-efficient fine-tuning (PEFT) for FL applications, elaborate on the readiness of FL frameworks to work with FMs, and provide future research opportunities on how to evaluate generative models in FL as well as the interplay of privacy and PEFT. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2401_04472 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | A Survey on Efficient Federated Learning Methods for Foundation Model Training Woisetschläger, Herbert Isenko, Alexander Wang, Shiqiang Mayer, Ruben Jacobsen, Hans-Arno Machine Learning Artificial Intelligence Distributed, Parallel, and Cluster Computing I.2.11; C.2 Federated Learning (FL) has become an established technique to facilitate privacy-preserving collaborative training across a multitude of clients. However, new approaches to FL often discuss their contributions involving small deep-learning models only and focus on training full models on clients. In the wake of Foundation Models (FM), the reality is different for many deep learning applications. Typically, FMs have already been pre-trained across a wide variety of tasks and can be fine-tuned to specific downstream tasks over significantly smaller datasets than required for full model training. However, access to such datasets is often challenging. By its design, FL can help to open data silos. With this survey, we introduce a novel taxonomy focused on computational and communication efficiency, the vital elements to make use of FMs in FL systems. We discuss the benefits and drawbacks of parameter-efficient fine-tuning (PEFT) for FL applications, elaborate on the readiness of FL frameworks to work with FMs, and provide future research opportunities on how to evaluate generative models in FL as well as the interplay of privacy and PEFT. |
| title | A Survey on Efficient Federated Learning Methods for Foundation Model Training |
| topic | Machine Learning Artificial Intelligence Distributed, Parallel, and Cluster Computing I.2.11; C.2 |
| url | https://arxiv.org/abs/2401.04472 |