Understanding Transferable Representation Learning and Zero-shot Transfer in CLIP
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2023
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866913425943166976 |
|---|---|
| author | Chen, Zixiang Deng, Yihe Li, Yuanzhi Gu, Quanquan |
| author_facet | Chen, Zixiang Deng, Yihe Li, Yuanzhi Gu, Quanquan |
| contents | Multi-modal learning has become increasingly popular due to its ability to leverage information from different data sources (e.g., text and images) to improve the model performance. Recently, CLIP has emerged as an effective approach that employs vision-language contrastive pretraining to learn joint image and text representations and exhibits remarkable performance in zero-shot learning and text-guided natural image generation. Despite the huge practical success of CLIP, its theoretical understanding remains elusive. In this paper, we formally study transferrable representation learning underlying CLIP and demonstrate how features from different modalities get aligned. We also analyze its zero-shot transfer performance on the downstream tasks. Inspired by our analysis, we propose a new CLIP-type approach, which achieves better performance than CLIP and other state-of-the-art methods on benchmark datasets. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2310_00927 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Understanding Transferable Representation Learning and Zero-shot Transfer in CLIP Chen, Zixiang Deng, Yihe Li, Yuanzhi Gu, Quanquan Machine Learning Artificial Intelligence Multi-modal learning has become increasingly popular due to its ability to leverage information from different data sources (e.g., text and images) to improve the model performance. Recently, CLIP has emerged as an effective approach that employs vision-language contrastive pretraining to learn joint image and text representations and exhibits remarkable performance in zero-shot learning and text-guided natural image generation. Despite the huge practical success of CLIP, its theoretical understanding remains elusive. In this paper, we formally study transferrable representation learning underlying CLIP and demonstrate how features from different modalities get aligned. We also analyze its zero-shot transfer performance on the downstream tasks. Inspired by our analysis, we propose a new CLIP-type approach, which achieves better performance than CLIP and other state-of-the-art methods on benchmark datasets. |
| title | Understanding Transferable Representation Learning and Zero-shot Transfer in CLIP |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2310.00927 |