Robust Federated Finetuning of LLMs via Alternating Optimization of LoRA
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912688044507136 |
|---|---|
| author | Chen, Shuangyi Guo, Yuanxin Ju, Yue Dalal, Harik Zhu, Zhongwen Khisti, Ashish |
| author_facet | Chen, Shuangyi Guo, Yuanxin Ju, Yue Dalal, Harik Zhu, Zhongwen Khisti, Ashish |
| contents | Parameter-Efficient Fine-Tuning (PEFT) methods like Low-Rank Adaptation (LoRA) optimize federated training by reducing computational and communication costs. We propose RoLoRA, a federated framework using alternating optimization to fine-tune LoRA adapters. Our approach emphasizes the importance of learning up and down projection matrices to enhance expressiveness and robustness. We use both theoretical analysis and extensive experiments to demonstrate the advantages of RoLoRA over prior approaches that either generate imperfect model updates or limit expressiveness of the model. We provide a theoretical analysis on a linear model to highlight the importance of learning both the down-projection and up-projection matrices in LoRA. We validate the insights on a non-linear model and separately provide a convergence proof under general conditions. To bridge theory and practice, we conducted extensive experimental evaluations on language models including RoBERTa-Large, Llama-2-7B on diverse tasks and FL settings to demonstrate the advantages of RoLoRA over other methods. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_01755 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Robust Federated Finetuning of LLMs via Alternating Optimization of LoRA Chen, Shuangyi Guo, Yuanxin Ju, Yue Dalal, Harik Zhu, Zhongwen Khisti, Ashish Machine Learning Artificial Intelligence Parameter-Efficient Fine-Tuning (PEFT) methods like Low-Rank Adaptation (LoRA) optimize federated training by reducing computational and communication costs. We propose RoLoRA, a federated framework using alternating optimization to fine-tune LoRA adapters. Our approach emphasizes the importance of learning up and down projection matrices to enhance expressiveness and robustness. We use both theoretical analysis and extensive experiments to demonstrate the advantages of RoLoRA over prior approaches that either generate imperfect model updates or limit expressiveness of the model. We provide a theoretical analysis on a linear model to highlight the importance of learning both the down-projection and up-projection matrices in LoRA. We validate the insights on a non-linear model and separately provide a convergence proof under general conditions. To bridge theory and practice, we conducted extensive experimental evaluations on language models including RoBERTa-Large, Llama-2-7B on diverse tasks and FL settings to demonstrate the advantages of RoLoRA over other methods. |
| title | Robust Federated Finetuning of LLMs via Alternating Optimization of LoRA |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2502.01755 |