TripletCLIP: Improving Compositional Reasoning of CLIP via Synthetic Vision-Language Negatives
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866909377700560896 |
|---|---|
| author | Patel, Maitreya Kusumba, Abhiram Cheng, Sheng Kim, Changhoon Gokhale, Tejas Baral, Chitta Yang, Yezhou |
| author_facet | Patel, Maitreya Kusumba, Abhiram Cheng, Sheng Kim, Changhoon Gokhale, Tejas Baral, Chitta Yang, Yezhou |
| contents | Contrastive Language-Image Pretraining (CLIP) models maximize the mutual information between text and visual modalities to learn representations. This makes the nature of the training data a significant factor in the efficacy of CLIP for downstream tasks. However, the lack of compositional diversity in contemporary image-text datasets limits the compositional reasoning ability of CLIP. We show that generating ``hard'' negative captions via in-context learning and synthesizing corresponding negative images with text-to-image generators offers a solution. We introduce a novel contrastive pre-training strategy that leverages these hard negative captions and images in an alternating fashion to train CLIP. We demonstrate that our method, named TripletCLIP, when applied to existing datasets such as CC3M and CC12M, enhances the compositional capabilities of CLIP, resulting in an absolute improvement of over 9% on the SugarCrepe benchmark on an equal computational budget, as well as improvements in zero-shot image classification and image retrieval. Our code, models, and data are available at: https://tripletclip.github.io |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_02545 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | TripletCLIP: Improving Compositional Reasoning of CLIP via Synthetic Vision-Language Negatives Patel, Maitreya Kusumba, Abhiram Cheng, Sheng Kim, Changhoon Gokhale, Tejas Baral, Chitta Yang, Yezhou Computer Vision and Pattern Recognition Computation and Language Contrastive Language-Image Pretraining (CLIP) models maximize the mutual information between text and visual modalities to learn representations. This makes the nature of the training data a significant factor in the efficacy of CLIP for downstream tasks. However, the lack of compositional diversity in contemporary image-text datasets limits the compositional reasoning ability of CLIP. We show that generating ``hard'' negative captions via in-context learning and synthesizing corresponding negative images with text-to-image generators offers a solution. We introduce a novel contrastive pre-training strategy that leverages these hard negative captions and images in an alternating fashion to train CLIP. We demonstrate that our method, named TripletCLIP, when applied to existing datasets such as CC3M and CC12M, enhances the compositional capabilities of CLIP, resulting in an absolute improvement of over 9% on the SugarCrepe benchmark on an equal computational budget, as well as improvements in zero-shot image classification and image retrieval. Our code, models, and data are available at: https://tripletclip.github.io |
| title | TripletCLIP: Improving Compositional Reasoning of CLIP via Synthetic Vision-Language Negatives |
| topic | Computer Vision and Pattern Recognition Computation and Language |
| url | https://arxiv.org/abs/2411.02545 |