Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2506.15307 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910036213628928 |
|---|---|
| author | Luo, Jinglong Zhang, Zhuo Zhang, Yehong Liu, Shiyu Dong, Ye Wang, Hui Yu, Yue Zhou, Xun Xu, Zenglin |
| author_facet | Luo, Jinglong Zhang, Zhuo Zhang, Yehong Liu, Shiyu Dong, Ye Wang, Hui Yu, Yue Zhou, Xun Xu, Zenglin |
| contents | Large Language Models (LLMs) have revolutionized numerous fields, yet their adaptation to specialized tasks in privacy-sensitive domains such as healthcare and finance remains constrained due to the scarcity of accessible training data caused by stringent privacy requirements. Secure Multi-party Computation (MPC)-based privacy-preserving machine learning provides theoretical guarantees for the privacy of model parameters and data. However, its application to LLMs has been predominantly limited to inference, as fine-tuning introduces significant efficiency challenges, particularly in backward propagation, optimizer, and self-attention operations. To address these challenges, we propose SecP-Tuning, the first MPC-based framework designed for efficient, privacy-preserving prompt tuning of LLMs. SecP-Tuning innovatively integrates Forward-only Tuning (FoT) through the ``data owner-server interaction" paradigm, effectively removing the need for privacy-preserving computations in backward propagation and optimization processes. Furthermore, it devises an efficient privacy-preserving Random Feature Attention (RFA), effectively mitigating the computational complexity of softmax-based self-attention and circumventing MPC-incompatible nonlinear operations. Experimental results demonstrate that, compared to full-Parameter Supervised Fine-Tuning (SFT) and gradient-based prompt tuning, SecP-Tuning achieves approximately 12x and 16x end-to-end acceleration, as well as 17x and 20x reductions in communication overhead, respectively. Moreover, it delivers performance comparable to gradient-based methods across multiple few-shot tasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_15307 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | SecP-Tuning: Efficient Privacy-Preserving Prompt Tuning for Large Language Models via MPC Luo, Jinglong Zhang, Zhuo Zhang, Yehong Liu, Shiyu Dong, Ye Wang, Hui Yu, Yue Zhou, Xun Xu, Zenglin Machine Learning Large Language Models (LLMs) have revolutionized numerous fields, yet their adaptation to specialized tasks in privacy-sensitive domains such as healthcare and finance remains constrained due to the scarcity of accessible training data caused by stringent privacy requirements. Secure Multi-party Computation (MPC)-based privacy-preserving machine learning provides theoretical guarantees for the privacy of model parameters and data. However, its application to LLMs has been predominantly limited to inference, as fine-tuning introduces significant efficiency challenges, particularly in backward propagation, optimizer, and self-attention operations. To address these challenges, we propose SecP-Tuning, the first MPC-based framework designed for efficient, privacy-preserving prompt tuning of LLMs. SecP-Tuning innovatively integrates Forward-only Tuning (FoT) through the ``data owner-server interaction" paradigm, effectively removing the need for privacy-preserving computations in backward propagation and optimization processes. Furthermore, it devises an efficient privacy-preserving Random Feature Attention (RFA), effectively mitigating the computational complexity of softmax-based self-attention and circumventing MPC-incompatible nonlinear operations. Experimental results demonstrate that, compared to full-Parameter Supervised Fine-Tuning (SFT) and gradient-based prompt tuning, SecP-Tuning achieves approximately 12x and 16x end-to-end acceleration, as well as 17x and 20x reductions in communication overhead, respectively. Moreover, it delivers performance comparable to gradient-based methods across multiple few-shot tasks. |
| title | SecP-Tuning: Efficient Privacy-Preserving Prompt Tuning for Large Language Models via MPC |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2506.15307 |