Characterizing the Training Dynamics of Private Fine-tuning with Langevin diffusion
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866915604646068224 |
|---|---|
| author | Ke, Shuqi Hou, Charlie Oh, Sewoong Fanti, Giulia |
| author_facet | Ke, Shuqi Hou, Charlie Oh, Sewoong Fanti, Giulia |
| contents | We show that differentially private full fine-tuning (DP-FFT) can distort pre-trained backbone features based on both theoretical and empirical results. We identify the cause of the distortion as the misalignment between the pre-trained backbone and the randomly initialized linear head. We prove that a sequential fine-tuning strategy can mitigate the feature distortion: first-linear-probing-then-fine-tuning (DP-LP-FFT). A new approximation scheme allows us to derive approximate upper and lower bounds on the training loss of DP-LP and DP-FFT, in a simple but canonical setting of 2-layer neural networks with ReLU activation. Experiments on real-world datasets and architectures are consistent with our theoretical insights. We also derive new upper bounds for 2-layer linear networks without the approximation. Moreover, our theory suggests a trade-off of privacy budget allocation in multi-phase fine-tuning methods like DP-LP-FFT. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_18905 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Characterizing the Training Dynamics of Private Fine-tuning with Langevin diffusion Ke, Shuqi Hou, Charlie Oh, Sewoong Fanti, Giulia Machine Learning Artificial Intelligence Cryptography and Security Optimization and Control We show that differentially private full fine-tuning (DP-FFT) can distort pre-trained backbone features based on both theoretical and empirical results. We identify the cause of the distortion as the misalignment between the pre-trained backbone and the randomly initialized linear head. We prove that a sequential fine-tuning strategy can mitigate the feature distortion: first-linear-probing-then-fine-tuning (DP-LP-FFT). A new approximation scheme allows us to derive approximate upper and lower bounds on the training loss of DP-LP and DP-FFT, in a simple but canonical setting of 2-layer neural networks with ReLU activation. Experiments on real-world datasets and architectures are consistent with our theoretical insights. We also derive new upper bounds for 2-layer linear networks without the approximation. Moreover, our theory suggests a trade-off of privacy budget allocation in multi-phase fine-tuning methods like DP-LP-FFT. |
| title | Characterizing the Training Dynamics of Private Fine-tuning with Langevin diffusion |
| topic | Machine Learning Artificial Intelligence Cryptography and Security Optimization and Control |
| url | https://arxiv.org/abs/2402.18905 |