Control-CLIP: Decoupling Category and Style Guidance in CLIP for Specific-Domain Generation
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866913692861333504 |
|---|---|
| author | Jia, Zexi Huang, Chuanwei Fei, Hongyan Zhu, Yeshuang Yuan, Zhiqiang Zhang, Jinchao Zhou, Jie |
| author_facet | Jia, Zexi Huang, Chuanwei Fei, Hongyan Zhu, Yeshuang Yuan, Zhiqiang Zhang, Jinchao Zhou, Jie |
| contents | Text-to-image diffusion models have shown remarkable capabilities of generating high-quality images closely aligned with textual inputs. However, the effectiveness of text guidance heavily relies on the CLIP text encoder, which is trained to pay more attention to general content but struggles to capture semantics in specific domains like styles. As a result, generation models tend to fail on prompts like "a photo of a cat in Pokemon style" in terms of simply producing images depicting "a photo of a cat". To fill this gap, we propose Control-CLIP, a novel decoupled CLIP fine-tuning framework that enables the CLIP model to learn the meaning of category and style in a complement manner. With specially designed fine-tuning tasks on minimal data and a modified cross-attention mechanism, Control-CLIP can precisely guide the diffusion model to a specific domain. Moreover, the parameters of the diffusion model remain unchanged at all, preserving the original generation performance and diversity. Experiments across multiple domains confirm the effectiveness of our approach, particularly highlighting its robust plug-and-play capability in generating content with various specific styles. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_11532 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Control-CLIP: Decoupling Category and Style Guidance in CLIP for Specific-Domain Generation Jia, Zexi Huang, Chuanwei Fei, Hongyan Zhu, Yeshuang Yuan, Zhiqiang Zhang, Jinchao Zhou, Jie Computer Vision and Pattern Recognition Text-to-image diffusion models have shown remarkable capabilities of generating high-quality images closely aligned with textual inputs. However, the effectiveness of text guidance heavily relies on the CLIP text encoder, which is trained to pay more attention to general content but struggles to capture semantics in specific domains like styles. As a result, generation models tend to fail on prompts like "a photo of a cat in Pokemon style" in terms of simply producing images depicting "a photo of a cat". To fill this gap, we propose Control-CLIP, a novel decoupled CLIP fine-tuning framework that enables the CLIP model to learn the meaning of category and style in a complement manner. With specially designed fine-tuning tasks on minimal data and a modified cross-attention mechanism, Control-CLIP can precisely guide the diffusion model to a specific domain. Moreover, the parameters of the diffusion model remain unchanged at all, preserving the original generation performance and diversity. Experiments across multiple domains confirm the effectiveness of our approach, particularly highlighting its robust plug-and-play capability in generating content with various specific styles. |
| title | Control-CLIP: Decoupling Category and Style Guidance in CLIP for Specific-Domain Generation |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2502.11532 |