You'll Never Walk Alone: A Sketch and Text Duet for Fine-Grained Image Retrieval
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866917618847318016 |
|---|---|
| author | Koley, Subhadeep Bhunia, Ayan Kumar Sain, Aneeshan Chowdhury, Pinaki Nath Xiang, Tao Song, Yi-Zhe |
| author_facet | Koley, Subhadeep Bhunia, Ayan Kumar Sain, Aneeshan Chowdhury, Pinaki Nath Xiang, Tao Song, Yi-Zhe |
| contents | Two primary input modalities prevail in image retrieval: sketch and text. While text is widely used for inter-category retrieval tasks, sketches have been established as the sole preferred modality for fine-grained image retrieval due to their ability to capture intricate visual details. In this paper, we question the reliance on sketches alone for fine-grained image retrieval by simultaneously exploring the fine-grained representation capabilities of both sketch and text, orchestrating a duet between the two. The end result enables precise retrievals previously unattainable, allowing users to pose ever-finer queries and incorporate attributes like colour and contextual cues from text. For this purpose, we introduce a novel compositionality framework, effectively combining sketches and text using pre-trained CLIP models, while eliminating the need for extensive fine-grained textual descriptions. Last but not least, our system extends to novel applications in composed image retrieval, domain attribute transfer, and fine-grained generation, providing solutions for various real-world scenarios. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_07222 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | You'll Never Walk Alone: A Sketch and Text Duet for Fine-Grained Image Retrieval Koley, Subhadeep Bhunia, Ayan Kumar Sain, Aneeshan Chowdhury, Pinaki Nath Xiang, Tao Song, Yi-Zhe Computer Vision and Pattern Recognition Two primary input modalities prevail in image retrieval: sketch and text. While text is widely used for inter-category retrieval tasks, sketches have been established as the sole preferred modality for fine-grained image retrieval due to their ability to capture intricate visual details. In this paper, we question the reliance on sketches alone for fine-grained image retrieval by simultaneously exploring the fine-grained representation capabilities of both sketch and text, orchestrating a duet between the two. The end result enables precise retrievals previously unattainable, allowing users to pose ever-finer queries and incorporate attributes like colour and contextual cues from text. For this purpose, we introduce a novel compositionality framework, effectively combining sketches and text using pre-trained CLIP models, while eliminating the need for extensive fine-grained textual descriptions. Last but not least, our system extends to novel applications in composed image retrieval, domain attribute transfer, and fine-grained generation, providing solutions for various real-world scenarios. |
| title | You'll Never Walk Alone: A Sketch and Text Duet for Fine-Grained Image Retrieval |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2403.07222 |