Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2502.04387 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915612369879040 |
|---|---|
| author | Lee, Royson Kim, Minyoung Rezk, Fady Li, Rui Venieris, Stylianos I. Hospedales, Timothy |
| author_facet | Lee, Royson Kim, Minyoung Rezk, Fady Li, Rui Venieris, Stylianos I. Hospedales, Timothy |
| contents | Federated learning (FL) has enabled the training of multilingual large language models (LLMs) on diverse and decentralized multilingual data, especially on low-resource languages. To improve client-specific performance, personalization via the use of parameter-efficient fine-tuning (PEFT) modules such as LoRA is common. This involves a personalization strategy (PS), such as the design of the PEFT adapter structures (e.g., in which layers to add LoRAs and what ranks) and choice of hyperparameters (e.g., learning rates) for fine-tuning. Instead of manual PS configuration, we propose FedP$^2$EFT, a federated learning-to-personalize method for multilingual LLMs in cross-device FL settings. Unlike most existing PEFT structure selection methods, which are prone to overfitting low-data regimes, FedP$^2$EFT collaboratively learns the optimal personalized PEFT structure for each client via Bayesian sparse rank selection. Evaluations on both simulated and real-world multilingual FL benchmarks demonstrate that FedP$^2$EFT largely outperforms existing personalized fine-tuning methods, while complementing other existing FL methods. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_04387 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | FedP$^2$EFT: Federated Learning to Personalize PEFT for Multilingual LLMs Lee, Royson Kim, Minyoung Rezk, Fady Li, Rui Venieris, Stylianos I. Hospedales, Timothy Computation and Language Artificial Intelligence Federated learning (FL) has enabled the training of multilingual large language models (LLMs) on diverse and decentralized multilingual data, especially on low-resource languages. To improve client-specific performance, personalization via the use of parameter-efficient fine-tuning (PEFT) modules such as LoRA is common. This involves a personalization strategy (PS), such as the design of the PEFT adapter structures (e.g., in which layers to add LoRAs and what ranks) and choice of hyperparameters (e.g., learning rates) for fine-tuning. Instead of manual PS configuration, we propose FedP$^2$EFT, a federated learning-to-personalize method for multilingual LLMs in cross-device FL settings. Unlike most existing PEFT structure selection methods, which are prone to overfitting low-data regimes, FedP$^2$EFT collaboratively learns the optimal personalized PEFT structure for each client via Bayesian sparse rank selection. Evaluations on both simulated and real-world multilingual FL benchmarks demonstrate that FedP$^2$EFT largely outperforms existing personalized fine-tuning methods, while complementing other existing FL methods. |
| title | FedP$^2$EFT: Federated Learning to Personalize PEFT for Multilingual LLMs |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2502.04387 |