It's All About Your Sketch: Democratising Sketch Control in Diffusion Models
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916169446850560 |
|---|---|
| author | Koley, Subhadeep Bhunia, Ayan Kumar Sekhri, Deeptanshu Sain, Aneeshan Chowdhury, Pinaki Nath Xiang, Tao Song, Yi-Zhe |
| author_facet | Koley, Subhadeep Bhunia, Ayan Kumar Sekhri, Deeptanshu Sain, Aneeshan Chowdhury, Pinaki Nath Xiang, Tao Song, Yi-Zhe |
| contents | This paper unravels the potential of sketches for diffusion models, addressing the deceptive promise of direct sketch control in generative AI. We importantly democratise the process, enabling amateur sketches to generate precise images, living up to the commitment of "what you sketch is what you get". A pilot study underscores the necessity, revealing that deformities in existing models stem from spatial-conditioning. To rectify this, we propose an abstraction-aware framework, utilising a sketch adapter, adaptive time-step sampling, and discriminative guidance from a pre-trained fine-grained sketch-based image retrieval model, working synergistically to reinforce fine-grained sketch-photo association. Our approach operates seamlessly during inference without the need for textual prompts; a simple, rough sketch akin to what you and I can create suffices! We welcome everyone to examine results presented in the paper and its supplementary. Contributions include democratising sketch control, introducing an abstraction-aware framework, and leveraging discriminative guidance, validated through extensive experiments. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_07234 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | It's All About Your Sketch: Democratising Sketch Control in Diffusion Models Koley, Subhadeep Bhunia, Ayan Kumar Sekhri, Deeptanshu Sain, Aneeshan Chowdhury, Pinaki Nath Xiang, Tao Song, Yi-Zhe Computer Vision and Pattern Recognition This paper unravels the potential of sketches for diffusion models, addressing the deceptive promise of direct sketch control in generative AI. We importantly democratise the process, enabling amateur sketches to generate precise images, living up to the commitment of "what you sketch is what you get". A pilot study underscores the necessity, revealing that deformities in existing models stem from spatial-conditioning. To rectify this, we propose an abstraction-aware framework, utilising a sketch adapter, adaptive time-step sampling, and discriminative guidance from a pre-trained fine-grained sketch-based image retrieval model, working synergistically to reinforce fine-grained sketch-photo association. Our approach operates seamlessly during inference without the need for textual prompts; a simple, rough sketch akin to what you and I can create suffices! We welcome everyone to examine results presented in the paper and its supplementary. Contributions include democratising sketch control, introducing an abstraction-aware framework, and leveraging discriminative guidance, validated through extensive experiments. |
| title | It's All About Your Sketch: Democratising Sketch Control in Diffusion Models |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2403.07234 |