DP-RDM: Adapting Diffusion Models to Private Domains Without Fine-Tuning
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866914792806023168 |
|---|---|
| author | Lebensold, Jonathan Sanjabi, Maziar Astolfi, Pietro Romero-Soriano, Adriana Chaudhuri, Kamalika Rabbat, Mike Guo, Chuan |
| author_facet | Lebensold, Jonathan Sanjabi, Maziar Astolfi, Pietro Romero-Soriano, Adriana Chaudhuri, Kamalika Rabbat, Mike Guo, Chuan |
| contents | Text-to-image diffusion models have been shown to suffer from sample-level memorization, possibly reproducing near-perfect replica of images that they are trained on, which may be undesirable. To remedy this issue, we develop the first differentially private (DP) retrieval-augmented generation algorithm that is capable of generating high-quality image samples while providing provable privacy guarantees. Specifically, we assume access to a text-to-image diffusion model trained on a small amount of public data, and design a DP retrieval mechanism to augment the text prompt with samples retrieved from a private retrieval dataset. Our \emph{differentially private retrieval-augmented diffusion model} (DP-RDM) requires no fine-tuning on the retrieval dataset to adapt to another domain, and can use state-of-the-art generative models to generate high-quality image samples while satisfying rigorous DP guarantees. For instance, when evaluated on MS-COCO, our DP-RDM can generate samples with a privacy budget of $ε=10$, while providing a $3.5$ point improvement in FID compared to public-only retrieval for up to $10,000$ queries. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_14421 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | DP-RDM: Adapting Diffusion Models to Private Domains Without Fine-Tuning Lebensold, Jonathan Sanjabi, Maziar Astolfi, Pietro Romero-Soriano, Adriana Chaudhuri, Kamalika Rabbat, Mike Guo, Chuan Machine Learning Cryptography and Security Computer Vision and Pattern Recognition Text-to-image diffusion models have been shown to suffer from sample-level memorization, possibly reproducing near-perfect replica of images that they are trained on, which may be undesirable. To remedy this issue, we develop the first differentially private (DP) retrieval-augmented generation algorithm that is capable of generating high-quality image samples while providing provable privacy guarantees. Specifically, we assume access to a text-to-image diffusion model trained on a small amount of public data, and design a DP retrieval mechanism to augment the text prompt with samples retrieved from a private retrieval dataset. Our \emph{differentially private retrieval-augmented diffusion model} (DP-RDM) requires no fine-tuning on the retrieval dataset to adapt to another domain, and can use state-of-the-art generative models to generate high-quality image samples while satisfying rigorous DP guarantees. For instance, when evaluated on MS-COCO, our DP-RDM can generate samples with a privacy budget of $ε=10$, while providing a $3.5$ point improvement in FID compared to public-only retrieval for up to $10,000$ queries. |
| title | DP-RDM: Adapting Diffusion Models to Private Domains Without Fine-Tuning |
| topic | Machine Learning Cryptography and Security Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2403.14421 |