Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2505.20808 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916973929037824 |
|---|---|
| author | Ruan, Bo-Kai Ni, Zi-Xiang Huang, Bo-Lun Hsiao, Teng-Fang Shuai, Hong-Han |
| author_facet | Ruan, Bo-Kai Ni, Zi-Xiang Huang, Bo-Lun Hsiao, Teng-Fang Shuai, Hong-Han |
| contents | Diffusion models achieve impressive performance in high-fidelity image generation but often struggle with rare concepts that appear infrequently in the training distribution. Prior work attempts to address this issue by prompt switching, where generation begins with a frequent proxy prompt and later transitions to the original rare prompt. However, such designs typically rely on fixed schedules that disregard the model's internal dynamics, making them brittle across prompts and backbones. In this paper, we re-frame rare prompt generation through the lens of score replacement: the denoising trajectory of a rare prompt can be initially guided by the score of a semantically related frequent prompt, which acts as a proxy. However, as the process unfolds, the proxy score gradually diverges from the true rare prompt score. To control this drift, we introduce a bounded deviation criterion that triggers the switch once the deviation exceeds a threshold. This formulation offers both a principled justification and a practical mechanism for rare prompt generation, enabling adaptive switching that can be widely adopted by different models. Extensive experiments across SDXL, SD3, Flux, and Sana confirm that our method consistently improves rare concept synthesis, outperforming strong baselines in both automated metrics and human evaluations. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_20808 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Score Replacement with Bounded Deviation for Rare Prompt Generation Ruan, Bo-Kai Ni, Zi-Xiang Huang, Bo-Lun Hsiao, Teng-Fang Shuai, Hong-Han Computer Vision and Pattern Recognition Diffusion models achieve impressive performance in high-fidelity image generation but often struggle with rare concepts that appear infrequently in the training distribution. Prior work attempts to address this issue by prompt switching, where generation begins with a frequent proxy prompt and later transitions to the original rare prompt. However, such designs typically rely on fixed schedules that disregard the model's internal dynamics, making them brittle across prompts and backbones. In this paper, we re-frame rare prompt generation through the lens of score replacement: the denoising trajectory of a rare prompt can be initially guided by the score of a semantically related frequent prompt, which acts as a proxy. However, as the process unfolds, the proxy score gradually diverges from the true rare prompt score. To control this drift, we introduce a bounded deviation criterion that triggers the switch once the deviation exceeds a threshold. This formulation offers both a principled justification and a practical mechanism for rare prompt generation, enabling adaptive switching that can be widely adopted by different models. Extensive experiments across SDXL, SD3, Flux, and Sana confirm that our method consistently improves rare concept synthesis, outperforming strong baselines in both automated metrics and human evaluations. |
| title | Score Replacement with Bounded Deviation for Rare Prompt Generation |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2505.20808 |