Aggregating Low Rank Adapters in Federated Fine-tuning
Fuente:
arXiv
Salvato in:
| Autori principali: | , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866918483229409280 |
|---|---|
| author | Trautmann, Evelyn Hales, Ian Volk, Martin F. |
| author_facet | Trautmann, Evelyn Hales, Ian Volk, Martin F. |
| contents | Fine-tuning large language models requires high computational and memory resources, and is therefore associated with significant costs. When training on federated datasets, an increased communication effort is also needed. For this reason, parameter-efficient methods (PEFT) are becoming increasingly important. In this context, very good results have already been achieved by fine-tuning with low-rank adaptation methods (LoRA). The application of LoRA methods in Federated Learning, and especially the aggregation of adaptation matrices, is a current research field. In this article, we propose a novel aggregation method and compare it with different existing aggregation methods of low rank adapters trained in a federated fine-tuning of large machine learning models and evaluate their performance with respect to selected GLUE benchmark datasets. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_06332 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Aggregating Low Rank Adapters in Federated Fine-tuning Trautmann, Evelyn Hales, Ian Volk, Martin F. Machine Learning Artificial Intelligence Fine-tuning large language models requires high computational and memory resources, and is therefore associated with significant costs. When training on federated datasets, an increased communication effort is also needed. For this reason, parameter-efficient methods (PEFT) are becoming increasingly important. In this context, very good results have already been achieved by fine-tuning with low-rank adaptation methods (LoRA). The application of LoRA methods in Federated Learning, and especially the aggregation of adaptation matrices, is a current research field. In this article, we propose a novel aggregation method and compare it with different existing aggregation methods of low rank adapters trained in a federated fine-tuning of large machine learning models and evaluate their performance with respect to selected GLUE benchmark datasets. |
| title | Aggregating Low Rank Adapters in Federated Fine-tuning |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2501.06332 |