Selective Pre-training for Private Fine-tuning
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2023
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866914855232995328 |
|---|---|
| author | Yu, Da Gopi, Sivakanth Kulkarni, Janardhan Lin, Zinan Naik, Saurabh Religa, Tomasz Lukasz Yin, Jian Zhang, Huishuai |
| author_facet | Yu, Da Gopi, Sivakanth Kulkarni, Janardhan Lin, Zinan Naik, Saurabh Religa, Tomasz Lukasz Yin, Jian Zhang, Huishuai |
| contents | Text prediction models, when used in applications like email clients or word processors, must protect user data privacy and adhere to model size constraints. These constraints are crucial to meet memory and inference time requirements, as well as to reduce inference costs. Building small, fast, and private domain-specific language models is a thriving area of research. In this work, we show that a careful pre-training on a \emph{subset} of the public dataset that is guided by the private dataset is crucial to train small language models with differential privacy. On standard benchmarks, small models trained with our new framework achieve state-of-the-art performance. In addition to performance improvements, our results demonstrate that smaller models, through careful pre-training and private fine-tuning, can match the performance of much larger models that do not have access to private data. This underscores the potential of private learning for model compression and enhanced efficiency. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2305_13865 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Selective Pre-training for Private Fine-tuning Yu, Da Gopi, Sivakanth Kulkarni, Janardhan Lin, Zinan Naik, Saurabh Religa, Tomasz Lukasz Yin, Jian Zhang, Huishuai Machine Learning Cryptography and Security Text prediction models, when used in applications like email clients or word processors, must protect user data privacy and adhere to model size constraints. These constraints are crucial to meet memory and inference time requirements, as well as to reduce inference costs. Building small, fast, and private domain-specific language models is a thriving area of research. In this work, we show that a careful pre-training on a \emph{subset} of the public dataset that is guided by the private dataset is crucial to train small language models with differential privacy. On standard benchmarks, small models trained with our new framework achieve state-of-the-art performance. In addition to performance improvements, our results demonstrate that smaller models, through careful pre-training and private fine-tuning, can match the performance of much larger models that do not have access to private data. This underscores the potential of private learning for model compression and enhanced efficiency. |
| title | Selective Pre-training for Private Fine-tuning |
| topic | Machine Learning Cryptography and Security |
| url | https://arxiv.org/abs/2305.13865 |