SCOT: Self-Supervised Contrastive Pretraining For Zero-Shot Compositional Retrieval
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866910785259700224 |
|---|---|
| author | Jawade, Bhavin Soares, Joao V. B. Thadani, Kapil Mohan, Deen Dayal Eshratifar, Amir Erfan Culpepper, Benjamin de Juan, Paloma Setlur, Srirangaraj Govindaraju, Venu |
| author_facet | Jawade, Bhavin Soares, Joao V. B. Thadani, Kapil Mohan, Deen Dayal Eshratifar, Amir Erfan Culpepper, Benjamin de Juan, Paloma Setlur, Srirangaraj Govindaraju, Venu |
| contents | Compositional image retrieval (CIR) is a multimodal learning task where a model combines a query image with a user-provided text modification to retrieve a target image. CIR finds applications in a variety of domains including product retrieval (e-commerce) and web search. Existing methods primarily focus on fully-supervised learning, wherein models are trained on datasets of labeled triplets such as FashionIQ and CIRR. This poses two significant challenges: (i) curating such triplet datasets is labor intensive; and (ii) models lack generalization to unseen objects and domains. In this work, we propose SCOT (Self-supervised COmpositional Training), a novel zero-shot compositional pretraining strategy that combines existing large image-text pair datasets with the generative capabilities of large language models to contrastively train an embedding composition network. Specifically, we show that the text embedding from a large-scale contrastively-pretrained vision-language model can be utilized as proxy target supervision during compositional pretraining, replacing the target image embedding. In zero-shot settings, this strategy surpasses SOTA zero-shot compositional retrieval methods as well as many fully-supervised methods on standard benchmarks such as FashionIQ and CIRR. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_08347 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | SCOT: Self-Supervised Contrastive Pretraining For Zero-Shot Compositional Retrieval Jawade, Bhavin Soares, Joao V. B. Thadani, Kapil Mohan, Deen Dayal Eshratifar, Amir Erfan Culpepper, Benjamin de Juan, Paloma Setlur, Srirangaraj Govindaraju, Venu Computer Vision and Pattern Recognition Artificial Intelligence Compositional image retrieval (CIR) is a multimodal learning task where a model combines a query image with a user-provided text modification to retrieve a target image. CIR finds applications in a variety of domains including product retrieval (e-commerce) and web search. Existing methods primarily focus on fully-supervised learning, wherein models are trained on datasets of labeled triplets such as FashionIQ and CIRR. This poses two significant challenges: (i) curating such triplet datasets is labor intensive; and (ii) models lack generalization to unseen objects and domains. In this work, we propose SCOT (Self-supervised COmpositional Training), a novel zero-shot compositional pretraining strategy that combines existing large image-text pair datasets with the generative capabilities of large language models to contrastively train an embedding composition network. Specifically, we show that the text embedding from a large-scale contrastively-pretrained vision-language model can be utilized as proxy target supervision during compositional pretraining, replacing the target image embedding. In zero-shot settings, this strategy surpasses SOTA zero-shot compositional retrieval methods as well as many fully-supervised methods on standard benchmarks such as FashionIQ and CIRR. |
| title | SCOT: Self-Supervised Contrastive Pretraining For Zero-Shot Compositional Retrieval |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence |
| url | https://arxiv.org/abs/2501.08347 |