ADF-LoRA: Alternating Low-Rank Aggregation for Decentralized Federated Fine-Tuning
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909919534383104 |
|---|---|
| author | Wang, Xiaoyu Li, Xiaotian Zhou, Zhixiang Li, Chen Liu, Yong |
| author_facet | Wang, Xiaoyu Li, Xiaotian Zhou, Zhixiang Li, Chen Liu, Yong |
| contents | This paper revisits alternating low-rank updates for federated fine-tuning and examines their behavior in decentralized federated learning (DFL). While alternating the LoRA matrices has been shown to stabilize aggregation in centralized FL, extending this mechanism to decentralized, peer-to-peer communication introduces new challenges due to phase-state mismatch and block-wise divergence across clients. We introduce ADF-LoRA, which synchronizes the update of only one low-rank matrix per round and mixes both matrices to maintain more consistent parameter states under decentralized propagation. This design preserves the cross-term suppression effect of alternating updates while improving stability in serverless topologies. We provide a convergence analysis under standard smoothness assumptions and evaluate ADF-LoRA on multiple GLUE tasks. Experiments show that ADF-LoRA achieves faster and smoother convergence and delivers the highest average accuracy across tasks, outperforming existing LoRA variants in decentralized FL by a consistent margin. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_18291 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | ADF-LoRA: Alternating Low-Rank Aggregation for Decentralized Federated Fine-Tuning Wang, Xiaoyu Li, Xiaotian Zhou, Zhixiang Li, Chen Liu, Yong Machine Learning Distributed, Parallel, and Cluster Computing I.2.11; I.2.6 This paper revisits alternating low-rank updates for federated fine-tuning and examines their behavior in decentralized federated learning (DFL). While alternating the LoRA matrices has been shown to stabilize aggregation in centralized FL, extending this mechanism to decentralized, peer-to-peer communication introduces new challenges due to phase-state mismatch and block-wise divergence across clients. We introduce ADF-LoRA, which synchronizes the update of only one low-rank matrix per round and mixes both matrices to maintain more consistent parameter states under decentralized propagation. This design preserves the cross-term suppression effect of alternating updates while improving stability in serverless topologies. We provide a convergence analysis under standard smoothness assumptions and evaluate ADF-LoRA on multiple GLUE tasks. Experiments show that ADF-LoRA achieves faster and smoother convergence and delivers the highest average accuracy across tasks, outperforming existing LoRA variants in decentralized FL by a consistent margin. |
| title | ADF-LoRA: Alternating Low-Rank Aggregation for Decentralized Federated Fine-Tuning |
| topic | Machine Learning Distributed, Parallel, and Cluster Computing I.2.11; I.2.6 |
| url | https://arxiv.org/abs/2511.18291 |