Capacity Control is an Effective Memorization Mitigation Mechanism in Text-Conditional Diffusion Models
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916459622432768 |
|---|---|
| author | Dutt, Raman Sanchez, Pedro Bohdal, Ondrej Tsaftaris, Sotirios A. Hospedales, Timothy |
| author_facet | Dutt, Raman Sanchez, Pedro Bohdal, Ondrej Tsaftaris, Sotirios A. Hospedales, Timothy |
| contents | In this work, we present compelling evidence that controlling model capacity during fine-tuning can effectively mitigate memorization in diffusion models. Specifically, we demonstrate that adopting Parameter-Efficient Fine-Tuning (PEFT) within the pre-train fine-tune paradigm significantly reduces memorization compared to traditional full fine-tuning approaches. Our experiments utilize the MIMIC dataset, which comprises image-text pairs of chest X-rays and their corresponding reports. The results, evaluated through a range of memorization and generation quality metrics, indicate that PEFT not only diminishes memorization but also enhances downstream generation quality. Additionally, PEFT methods can be seamlessly combined with existing memorization mitigation techniques for further improvement. The code for our experiments is available at: https://github.com/Raman1121/Diffusion_Memorization_HPO |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_22149 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Capacity Control is an Effective Memorization Mitigation Mechanism in Text-Conditional Diffusion Models Dutt, Raman Sanchez, Pedro Bohdal, Ondrej Tsaftaris, Sotirios A. Hospedales, Timothy Computer Vision and Pattern Recognition In this work, we present compelling evidence that controlling model capacity during fine-tuning can effectively mitigate memorization in diffusion models. Specifically, we demonstrate that adopting Parameter-Efficient Fine-Tuning (PEFT) within the pre-train fine-tune paradigm significantly reduces memorization compared to traditional full fine-tuning approaches. Our experiments utilize the MIMIC dataset, which comprises image-text pairs of chest X-rays and their corresponding reports. The results, evaluated through a range of memorization and generation quality metrics, indicate that PEFT not only diminishes memorization but also enhances downstream generation quality. Additionally, PEFT methods can be seamlessly combined with existing memorization mitigation techniques for further improvement. The code for our experiments is available at: https://github.com/Raman1121/Diffusion_Memorization_HPO |
| title | Capacity Control is an Effective Memorization Mitigation Mechanism in Text-Conditional Diffusion Models |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2410.22149 |