Encryption-Friendly LLM Architecture
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866912239647195136 |
|---|---|
| author | Rho, Donghwan Kim, Taeseong Park, Minje Kim, Jung Woo Chae, Hyunsik Ryu, Ernest K. Cheon, Jung Hee |
| author_facet | Rho, Donghwan Kim, Taeseong Park, Minje Kim, Jung Woo Chae, Hyunsik Ryu, Ernest K. Cheon, Jung Hee |
| contents | Large language models (LLMs) offer personalized responses based on user interactions, but this use case raises serious privacy concerns. Homomorphic encryption (HE) is a cryptographic protocol supporting arithmetic computations in encrypted states and provides a potential solution for privacy-preserving machine learning (PPML). However, the computational intensity of transformers poses challenges for applying HE to LLMs. In this work, we propose a modified HE-friendly transformer architecture with an emphasis on inference following personalized (private) fine-tuning. Utilizing LoRA fine-tuning and Gaussian kernels, we achieve significant computational speedups -- 6.94x for fine-tuning and 2.3x for inference -- while maintaining performance comparable to plaintext models. Our findings provide a viable proof of concept for offering privacy-preserving LLM services in areas where data protection is crucial. Our code is available on GitHub. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_02486 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Encryption-Friendly LLM Architecture Rho, Donghwan Kim, Taeseong Park, Minje Kim, Jung Woo Chae, Hyunsik Ryu, Ernest K. Cheon, Jung Hee Cryptography and Security Machine Learning Large language models (LLMs) offer personalized responses based on user interactions, but this use case raises serious privacy concerns. Homomorphic encryption (HE) is a cryptographic protocol supporting arithmetic computations in encrypted states and provides a potential solution for privacy-preserving machine learning (PPML). However, the computational intensity of transformers poses challenges for applying HE to LLMs. In this work, we propose a modified HE-friendly transformer architecture with an emphasis on inference following personalized (private) fine-tuning. Utilizing LoRA fine-tuning and Gaussian kernels, we achieve significant computational speedups -- 6.94x for fine-tuning and 2.3x for inference -- while maintaining performance comparable to plaintext models. Our findings provide a viable proof of concept for offering privacy-preserving LLM services in areas where data protection is crucial. Our code is available on GitHub. |
| title | Encryption-Friendly LLM Architecture |
| topic | Cryptography and Security Machine Learning |
| url | https://arxiv.org/abs/2410.02486 |