Gespeichert in:
| Hauptverfasser: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | https://arxiv.org/abs/2507.04455 |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866915373422477312 |
|---|---|
| author | Yao, Kai Tan, Zhaorui Gao, Penglei Li, Lichun Wu, Kaixin Wang, Yinggui Zhao, Yuan Ji, Yixin Wang, Wei Zhu, Jianke |
| author_facet | Yao, Kai Tan, Zhaorui Gao, Penglei Li, Lichun Wu, Kaixin Wang, Yinggui Zhao, Yuan Ji, Yixin Wang, Wei Zhu, Jianke |
| contents | The rapid growth of large language models (LLMs) with traditional centralized fine-tuning emerges as a key technique for adapting these models to domain-specific challenges, yielding privacy risks for both model and data owners. One promising solution, called offsite-tuning (OT), is proposed to address these challenges, where a weaker emulator is compressed from the original model and further fine-tuned with adapter to enhance privacy. However, the existing OT-based methods require high computational costs and lack theoretical analysis. This paper introduces a novel OT approach based on gradient-preserving compression, named GradOT. By analyzing the OT problem through the lens of optimization, we propose a method that selectively applies compression techniques such as rank compression and channel pruning, preserving the gradients of fine-tuned adapters while ensuring privacy. Extensive experiments demonstrate that our approach surpasses existing OT methods, both in terms of privacy protection and model performance. Our method provides a theoretical foundation for OT and offers a practical, training-free solution for offsite-tuning of large-scale LLMs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_04455 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | GradOT: Training-free Gradient-preserving Offsite-tuning for Large Language Models Yao, Kai Tan, Zhaorui Gao, Penglei Li, Lichun Wu, Kaixin Wang, Yinggui Zhao, Yuan Ji, Yixin Wang, Wei Zhu, Jianke Computation and Language The rapid growth of large language models (LLMs) with traditional centralized fine-tuning emerges as a key technique for adapting these models to domain-specific challenges, yielding privacy risks for both model and data owners. One promising solution, called offsite-tuning (OT), is proposed to address these challenges, where a weaker emulator is compressed from the original model and further fine-tuned with adapter to enhance privacy. However, the existing OT-based methods require high computational costs and lack theoretical analysis. This paper introduces a novel OT approach based on gradient-preserving compression, named GradOT. By analyzing the OT problem through the lens of optimization, we propose a method that selectively applies compression techniques such as rank compression and channel pruning, preserving the gradients of fine-tuned adapters while ensuring privacy. Extensive experiments demonstrate that our approach surpasses existing OT methods, both in terms of privacy protection and model performance. Our method provides a theoretical foundation for OT and offers a practical, training-free solution for offsite-tuning of large-scale LLMs. |
| title | GradOT: Training-free Gradient-preserving Offsite-tuning for Large Language Models |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2507.04455 |