Enhancing Prompt Following with Visual Control Through Training-Free Mask-Guided Diffusion
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866910418750930944 |
|---|---|
| author | Chen, Hongyu Gao, Yiqi Zhou, Min Wang, Peng Li, Xubin Ge, Tiezheng Zheng, Bo |
| author_facet | Chen, Hongyu Gao, Yiqi Zhou, Min Wang, Peng Li, Xubin Ge, Tiezheng Zheng, Bo |
| contents | Recently, integrating visual controls into text-to-image~(T2I) models, such as ControlNet method, has received significant attention for finer control capabilities. While various training-free methods make efforts to enhance prompt following in T2I models, the issue with visual control is still rarely studied, especially in the scenario that visual controls are misaligned with text prompts. In this paper, we address the challenge of ``Prompt Following With Visual Control" and propose a training-free approach named Mask-guided Prompt Following (MGPF). Object masks are introduced to distinct aligned and misaligned parts of visual controls and prompts. Meanwhile, a network, dubbed as Masked ControlNet, is designed to utilize these object masks for object generation in the misaligned visual control region. Further, to improve attribute matching, a simple yet efficient loss is designed to align the attention maps of attributes with object regions constrained by ControlNet and object masks. The efficacy and superiority of MGPF are validated through comprehensive quantitative and qualitative experiments. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_14768 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Enhancing Prompt Following with Visual Control Through Training-Free Mask-Guided Diffusion Chen, Hongyu Gao, Yiqi Zhou, Min Wang, Peng Li, Xubin Ge, Tiezheng Zheng, Bo Computer Vision and Pattern Recognition Recently, integrating visual controls into text-to-image~(T2I) models, such as ControlNet method, has received significant attention for finer control capabilities. While various training-free methods make efforts to enhance prompt following in T2I models, the issue with visual control is still rarely studied, especially in the scenario that visual controls are misaligned with text prompts. In this paper, we address the challenge of ``Prompt Following With Visual Control" and propose a training-free approach named Mask-guided Prompt Following (MGPF). Object masks are introduced to distinct aligned and misaligned parts of visual controls and prompts. Meanwhile, a network, dubbed as Masked ControlNet, is designed to utilize these object masks for object generation in the misaligned visual control region. Further, to improve attribute matching, a simple yet efficient loss is designed to align the attention maps of attributes with object regions constrained by ControlNet and object masks. The efficacy and superiority of MGPF are validated through comprehensive quantitative and qualitative experiments. |
| title | Enhancing Prompt Following with Visual Control Through Training-Free Mask-Guided Diffusion |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2404.14768 |