Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2406.10976 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916288918454272 |
|---|---|
| author | Zhu, JianHao Lv, Changze Wang, Xiaohua Wu, Muling Liu, Wenhao Li, Tianlong Ling, Zixuan Zhang, Cenyuan Zheng, Xiaoqing Huang, Xuanjing |
| author_facet | Zhu, JianHao Lv, Changze Wang, Xiaohua Wu, Muling Liu, Wenhao Li, Tianlong Ling, Zixuan Zhang, Cenyuan Zheng, Xiaoqing Huang, Xuanjing |
| contents | Conventional federated learning primarily aims to secure the privacy of data distributed across multiple edge devices, with the global model dispatched to edge devices for parameter updates during the learning process. However, the development of large language models (LLMs) requires substantial data and computational resources, rendering them valuable intellectual properties for their developers and owners. To establish a mechanism that protects both data and model privacy in a federated learning context, we introduce a method that just needs to distribute a quantized version of the model's parameters during training. This method enables accurate gradient estimations for parameter updates while preventing clients from accessing a model whose performance is comparable to the centrally hosted one. Moreover, we combine this quantization strategy with LoRA, a popular and parameter-efficient fine-tuning method, to significantly reduce communication costs in federated learning. The proposed framework, named \textsc{FedLPP}, successfully ensures both data and model privacy in the federated learning context. Additionally, the learned central model exhibits good generalization and can be trained in a resource-efficient manner. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_10976 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Promoting Data and Model Privacy in Federated Learning through Quantized LoRA Zhu, JianHao Lv, Changze Wang, Xiaohua Wu, Muling Liu, Wenhao Li, Tianlong Ling, Zixuan Zhang, Cenyuan Zheng, Xiaoqing Huang, Xuanjing Machine Learning Computation and Language Cryptography and Security Conventional federated learning primarily aims to secure the privacy of data distributed across multiple edge devices, with the global model dispatched to edge devices for parameter updates during the learning process. However, the development of large language models (LLMs) requires substantial data and computational resources, rendering them valuable intellectual properties for their developers and owners. To establish a mechanism that protects both data and model privacy in a federated learning context, we introduce a method that just needs to distribute a quantized version of the model's parameters during training. This method enables accurate gradient estimations for parameter updates while preventing clients from accessing a model whose performance is comparable to the centrally hosted one. Moreover, we combine this quantization strategy with LoRA, a popular and parameter-efficient fine-tuning method, to significantly reduce communication costs in federated learning. The proposed framework, named \textsc{FedLPP}, successfully ensures both data and model privacy in the federated learning context. Additionally, the learned central model exhibits good generalization and can be trained in a resource-efficient manner. |
| title | Promoting Data and Model Privacy in Federated Learning through Quantized LoRA |
| topic | Machine Learning Computation and Language Cryptography and Security |
| url | https://arxiv.org/abs/2406.10976 |