Noise Consistency Regularization for Improved Subject-Driven Image Synthesis
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910993146183680 |
|---|---|
| author | Ni, Yao Wen, Song Koniusz, Piotr Cherian, Anoop |
| author_facet | Ni, Yao Wen, Song Koniusz, Piotr Cherian, Anoop |
| contents | Fine-tuning Stable Diffusion enables subject-driven image synthesis by adapting the model to generate images containing specific subjects. However, existing fine-tuning methods suffer from two key issues: underfitting, where the model fails to reliably capture subject identity, and overfitting, where it memorizes the subject image and reduces background diversity. To address these challenges, we propose two auxiliary consistency losses for diffusion fine-tuning. First, a prior consistency regularization loss ensures that the predicted diffusion noise for prior (non-subject) images remains consistent with that of the pretrained model, improving fidelity. Second, a subject consistency regularization loss enhances the fine-tuned model's robustness to multiplicative noise modulated latent code, helping to preserve subject identity while improving diversity. Our experimental results demonstrate that incorporating these losses into fine-tuning not only preserves subject identity but also enhances image diversity, outperforming DreamBooth in terms of CLIP scores, background variation, and overall visual quality. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_06483 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Noise Consistency Regularization for Improved Subject-Driven Image Synthesis Ni, Yao Wen, Song Koniusz, Piotr Cherian, Anoop Graphics Artificial Intelligence Computer Vision and Pattern Recognition Machine Learning Image and Video Processing Fine-tuning Stable Diffusion enables subject-driven image synthesis by adapting the model to generate images containing specific subjects. However, existing fine-tuning methods suffer from two key issues: underfitting, where the model fails to reliably capture subject identity, and overfitting, where it memorizes the subject image and reduces background diversity. To address these challenges, we propose two auxiliary consistency losses for diffusion fine-tuning. First, a prior consistency regularization loss ensures that the predicted diffusion noise for prior (non-subject) images remains consistent with that of the pretrained model, improving fidelity. Second, a subject consistency regularization loss enhances the fine-tuned model's robustness to multiplicative noise modulated latent code, helping to preserve subject identity while improving diversity. Our experimental results demonstrate that incorporating these losses into fine-tuning not only preserves subject identity but also enhances image diversity, outperforming DreamBooth in terms of CLIP scores, background variation, and overall visual quality. |
| title | Noise Consistency Regularization for Improved Subject-Driven Image Synthesis |
| topic | Graphics Artificial Intelligence Computer Vision and Pattern Recognition Machine Learning Image and Video Processing |
| url | https://arxiv.org/abs/2506.06483 |