সংরক্ষণ করুন:
| প্রধান লেখক: | , , , , |
|---|---|
| বিন্যাস: | Preprint |
| প্রকাশিত: |
2024
|
| বিষয়গুলি: | |
| অনলাইন ব্যবহার করুন: | https://arxiv.org/abs/2411.14639 |
| ট্যাগগুলো: |
ট্যাগ যুক্ত করুন
কোনো ট্যাগ নেই, প্রথমজন হিসাবে ট্যাগ করুন!
|
| _version_ | 1866918325954543616 |
|---|---|
| author | Peetathawatchai, Pura Chen, Wei-Ning Isik, Berivan Koyejo, Sanmi No, Albert |
| author_facet | Peetathawatchai, Pura Chen, Wei-Ning Isik, Berivan Koyejo, Sanmi No, Albert |
| contents | Personalizing large-scale diffusion models poses serious privacy risks, especially when adapting to small, sensitive datasets. A common approach is to fine-tune the model using differentially private stochastic gradient descent (DP-SGD), but this suffers from severe utility degradation due to the high noise needed for privacy, particularly in the small data regime. We propose an alternative that leverages Textual Inversion (TI), which learns an embedding vector for an image or set of images, to enable adaptation under differential privacy (DP) constraints. Our approach, Differentially Private Aggregation via Textual Inversion (DPAgg-TI), adds calibrated noise to the aggregation of per-image embeddings to ensure formal DP guarantees while preserving high output fidelity. We show that DPAgg-TI outperforms DP-SGD finetuning in both utility and robustness under the same privacy budget, achieving results closely matching the non-private baseline on style adaptation tasks using private artwork from a single artist and Paris 2024 Olympic pictograms. In contrast, DP-SGD fails to generate meaningful outputs in this setting. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_14639 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Differentially Private Adaptation of Diffusion Models via Noisy Aggregated Embeddings Peetathawatchai, Pura Chen, Wei-Ning Isik, Berivan Koyejo, Sanmi No, Albert Computer Vision and Pattern Recognition Cryptography and Security Machine Learning Computer Vision (cs.CV), Machine Learning (cs.LG), Machine Learning (stat.ML) Personalizing large-scale diffusion models poses serious privacy risks, especially when adapting to small, sensitive datasets. A common approach is to fine-tune the model using differentially private stochastic gradient descent (DP-SGD), but this suffers from severe utility degradation due to the high noise needed for privacy, particularly in the small data regime. We propose an alternative that leverages Textual Inversion (TI), which learns an embedding vector for an image or set of images, to enable adaptation under differential privacy (DP) constraints. Our approach, Differentially Private Aggregation via Textual Inversion (DPAgg-TI), adds calibrated noise to the aggregation of per-image embeddings to ensure formal DP guarantees while preserving high output fidelity. We show that DPAgg-TI outperforms DP-SGD finetuning in both utility and robustness under the same privacy budget, achieving results closely matching the non-private baseline on style adaptation tasks using private artwork from a single artist and Paris 2024 Olympic pictograms. In contrast, DP-SGD fails to generate meaningful outputs in this setting. |
| title | Differentially Private Adaptation of Diffusion Models via Noisy Aggregated Embeddings |
| topic | Computer Vision and Pattern Recognition Cryptography and Security Machine Learning Computer Vision (cs.CV), Machine Learning (cs.LG), Machine Learning (stat.ML) |
| url | https://arxiv.org/abs/2411.14639 |