DPZero: Private Fine-Tuning of Language Models without Backpropagation
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2023
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866909217838858240 |
|---|---|
| author | Zhang, Liang Li, Bingcong Thekumparampil, Kiran Koshy Oh, Sewoong He, Niao |
| author_facet | Zhang, Liang Li, Bingcong Thekumparampil, Kiran Koshy Oh, Sewoong He, Niao |
| contents | The widespread practice of fine-tuning large language models (LLMs) on domain-specific data faces two major challenges in memory and privacy. First, as the size of LLMs continues to grow, the memory demands of gradient-based training methods via backpropagation become prohibitively high. Second, given the tendency of LLMs to memorize training data, it is important to protect potentially sensitive information in the fine-tuning data from being regurgitated. Zeroth-order methods, which rely solely on forward passes, substantially reduce memory consumption during training. However, directly combining them with standard differentially private gradient descent suffers more as model size grows. To bridge this gap, we introduce DPZero, a novel private zeroth-order algorithm with nearly dimension-independent rates. The memory efficiency of DPZero is demonstrated in privately fine-tuning RoBERTa and OPT on several downstream tasks. Our code is available at https://github.com/Liang137/DPZero. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2310_09639 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | DPZero: Private Fine-Tuning of Language Models without Backpropagation Zhang, Liang Li, Bingcong Thekumparampil, Kiran Koshy Oh, Sewoong He, Niao Machine Learning Cryptography and Security Optimization and Control The widespread practice of fine-tuning large language models (LLMs) on domain-specific data faces two major challenges in memory and privacy. First, as the size of LLMs continues to grow, the memory demands of gradient-based training methods via backpropagation become prohibitively high. Second, given the tendency of LLMs to memorize training data, it is important to protect potentially sensitive information in the fine-tuning data from being regurgitated. Zeroth-order methods, which rely solely on forward passes, substantially reduce memory consumption during training. However, directly combining them with standard differentially private gradient descent suffers more as model size grows. To bridge this gap, we introduce DPZero, a novel private zeroth-order algorithm with nearly dimension-independent rates. The memory efficiency of DPZero is demonstrated in privately fine-tuning RoBERTa and OPT on several downstream tasks. Our code is available at https://github.com/Liang137/DPZero. |
| title | DPZero: Private Fine-Tuning of Language Models without Backpropagation |
| topic | Machine Learning Cryptography and Security Optimization and Control |
| url | https://arxiv.org/abs/2310.09639 |