FedsLLM: Federated Split Learning for Large Language Models over Communication Networks
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866911953612439552 |
|---|---|
| author | Zhao, Kai Yang, Zhaohui Huang, Chongwen Chen, Xiaoming Zhang, Zhaoyang |
| author_facet | Zhao, Kai Yang, Zhaohui Huang, Chongwen Chen, Xiaoming Zhang, Zhaoyang |
| contents | Addressing the challenges of deploying large language models in wireless communication networks, this paper combines low-rank adaptation technology (LoRA) with the splitfed learning framework to propose the federated split learning for large language models (FedsLLM) framework. The method introduced in this paper utilizes LoRA technology to reduce processing loads by dividing the network into client subnetworks and server subnetworks. It leverages a federated server to aggregate and update client models. As the training data are transmitted through a wireless network between clients and both main and federated servers, the training delay is determined by the learning accuracy and the allocation of communication bandwidth. This paper models the minimization of the training delay by integrating computation and communication optimization, simplifying the optimization problem into a convex problem to find the optimal solution. Additionally, it presents a lemma that describes the precise solutions to this problem. Simulation results demonstrate that the proposed optimization algorithm reduces delays by an average of 47.63% compared to unoptimized scenarios. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_09250 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | FedsLLM: Federated Split Learning for Large Language Models over Communication Networks Zhao, Kai Yang, Zhaohui Huang, Chongwen Chen, Xiaoming Zhang, Zhaoyang Networking and Internet Architecture Machine Learning Addressing the challenges of deploying large language models in wireless communication networks, this paper combines low-rank adaptation technology (LoRA) with the splitfed learning framework to propose the federated split learning for large language models (FedsLLM) framework. The method introduced in this paper utilizes LoRA technology to reduce processing loads by dividing the network into client subnetworks and server subnetworks. It leverages a federated server to aggregate and update client models. As the training data are transmitted through a wireless network between clients and both main and federated servers, the training delay is determined by the learning accuracy and the allocation of communication bandwidth. This paper models the minimization of the training delay by integrating computation and communication optimization, simplifying the optimization problem into a convex problem to find the optimal solution. Additionally, it presents a lemma that describes the precise solutions to this problem. Simulation results demonstrate that the proposed optimization algorithm reduces delays by an average of 47.63% compared to unoptimized scenarios. |
| title | FedsLLM: Federated Split Learning for Large Language Models over Communication Networks |
| topic | Networking and Internet Architecture Machine Learning |
| url | https://arxiv.org/abs/2407.09250 |