Fine-Tuning Language Models with Differential Privacy through Adaptive Noise Allocation
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916422793297920 |
|---|---|
| author | Li, Xianzhi Zmigrod, Ran Ma, Zhiqiang Liu, Xiaomo Zhu, Xiaodan |
| author_facet | Li, Xianzhi Zmigrod, Ran Ma, Zhiqiang Liu, Xiaomo Zhu, Xiaodan |
| contents | Language models are capable of memorizing detailed patterns and information, leading to a double-edged effect: they achieve impressive modeling performance on downstream tasks with the stored knowledge but also raise significant privacy concerns. Traditional differential privacy based training approaches offer robust safeguards by employing a uniform noise distribution across all parameters. However, this overlooks the distinct sensitivities and contributions of individual parameters in privacy protection and often results in suboptimal models. To address these limitations, we propose ANADP, a novel algorithm that adaptively allocates additive noise based on the importance of model parameters. We demonstrate that ANADP narrows the performance gap between regular fine-tuning and traditional DP fine-tuning on a series of datasets while maintaining the required privacy constraints. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_02912 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Fine-Tuning Language Models with Differential Privacy through Adaptive Noise Allocation Li, Xianzhi Zmigrod, Ran Ma, Zhiqiang Liu, Xiaomo Zhu, Xiaodan Artificial Intelligence Computation and Language Cryptography and Security Machine Learning Language models are capable of memorizing detailed patterns and information, leading to a double-edged effect: they achieve impressive modeling performance on downstream tasks with the stored knowledge but also raise significant privacy concerns. Traditional differential privacy based training approaches offer robust safeguards by employing a uniform noise distribution across all parameters. However, this overlooks the distinct sensitivities and contributions of individual parameters in privacy protection and often results in suboptimal models. To address these limitations, we propose ANADP, a novel algorithm that adaptively allocates additive noise based on the importance of model parameters. We demonstrate that ANADP narrows the performance gap between regular fine-tuning and traditional DP fine-tuning on a series of datasets while maintaining the required privacy constraints. |
| title | Fine-Tuning Language Models with Differential Privacy through Adaptive Noise Allocation |
| topic | Artificial Intelligence Computation and Language Cryptography and Security Machine Learning |
| url | https://arxiv.org/abs/2410.02912 |