FedRDMA: Communication-Efficient Cross-Silo Federated LLM via Chunked RDMA Transmission
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917602518892544 |
|---|---|
| author | Zhang, Zeling Cai, Dongqi Zhang, Yiran Xu, Mengwei Wang, Shangguang Zhou, Ao |
| author_facet | Zhang, Zeling Cai, Dongqi Zhang, Yiran Xu, Mengwei Wang, Shangguang Zhou, Ao |
| contents | Communication overhead is a significant bottleneck in federated learning (FL), which has been exaggerated with the increasing size of AI models. In this paper, we propose FedRDMA, a communication-efficient cross-silo FL system that integrates RDMA into the FL communication protocol. To overcome the limitations of RDMA in wide-area networks (WANs), FedRDMA divides the updated model into chunks and designs a series of optimization techniques to improve the efficiency and robustness of RDMA-based communication. We implement FedRDMA atop the industrial federated learning framework and evaluate it on a real-world cross-silo FL scenario. The experimental results show that \sys can achieve up to 3.8$\times$ speedup in communication efficiency compared to traditional TCP/IP-based FL systems. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_00881 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | FedRDMA: Communication-Efficient Cross-Silo Federated LLM via Chunked RDMA Transmission Zhang, Zeling Cai, Dongqi Zhang, Yiran Xu, Mengwei Wang, Shangguang Zhou, Ao Machine Learning Distributed, Parallel, and Cluster Computing Networking and Internet Architecture Communication overhead is a significant bottleneck in federated learning (FL), which has been exaggerated with the increasing size of AI models. In this paper, we propose FedRDMA, a communication-efficient cross-silo FL system that integrates RDMA into the FL communication protocol. To overcome the limitations of RDMA in wide-area networks (WANs), FedRDMA divides the updated model into chunks and designs a series of optimization techniques to improve the efficiency and robustness of RDMA-based communication. We implement FedRDMA atop the industrial federated learning framework and evaluate it on a real-world cross-silo FL scenario. The experimental results show that \sys can achieve up to 3.8$\times$ speedup in communication efficiency compared to traditional TCP/IP-based FL systems. |
| title | FedRDMA: Communication-Efficient Cross-Silo Federated LLM via Chunked RDMA Transmission |
| topic | Machine Learning Distributed, Parallel, and Cluster Computing Networking and Internet Architecture |
| url | https://arxiv.org/abs/2403.00881 |