Text-to-Image Diffusion Models are Great Sketch-Photo Matchmakers
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909144376672256 |
|---|---|
| author | Koley, Subhadeep Bhunia, Ayan Kumar Sain, Aneeshan Chowdhury, Pinaki Nath Xiang, Tao Song, Yi-Zhe |
| author_facet | Koley, Subhadeep Bhunia, Ayan Kumar Sain, Aneeshan Chowdhury, Pinaki Nath Xiang, Tao Song, Yi-Zhe |
| contents | This paper, for the first time, explores text-to-image diffusion models for Zero-Shot Sketch-based Image Retrieval (ZS-SBIR). We highlight a pivotal discovery: the capacity of text-to-image diffusion models to seamlessly bridge the gap between sketches and photos. This proficiency is underpinned by their robust cross-modal capabilities and shape bias, findings that are substantiated through our pilot studies. In order to harness pre-trained diffusion models effectively, we introduce a straightforward yet powerful strategy focused on two key aspects: selecting optimal feature layers and utilising visual and textual prompts. For the former, we identify which layers are most enriched with information and are best suited for the specific retrieval requirements (category-level or fine-grained). Then we employ visual and textual prompts to guide the model's feature extraction process, enabling it to generate more discriminative and contextually relevant cross-modal representations. Extensive experiments on several benchmark datasets validate significant performance improvements. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_07214 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Text-to-Image Diffusion Models are Great Sketch-Photo Matchmakers Koley, Subhadeep Bhunia, Ayan Kumar Sain, Aneeshan Chowdhury, Pinaki Nath Xiang, Tao Song, Yi-Zhe Computer Vision and Pattern Recognition This paper, for the first time, explores text-to-image diffusion models for Zero-Shot Sketch-based Image Retrieval (ZS-SBIR). We highlight a pivotal discovery: the capacity of text-to-image diffusion models to seamlessly bridge the gap between sketches and photos. This proficiency is underpinned by their robust cross-modal capabilities and shape bias, findings that are substantiated through our pilot studies. In order to harness pre-trained diffusion models effectively, we introduce a straightforward yet powerful strategy focused on two key aspects: selecting optimal feature layers and utilising visual and textual prompts. For the former, we identify which layers are most enriched with information and are best suited for the specific retrieval requirements (category-level or fine-grained). Then we employ visual and textual prompts to guide the model's feature extraction process, enabling it to generate more discriminative and contextually relevant cross-modal representations. Extensive experiments on several benchmark datasets validate significant performance improvements. |
| title | Text-to-Image Diffusion Models are Great Sketch-Photo Matchmakers |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2403.07214 |