Guardado en:
| Autores principales: | , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | https://arxiv.org/abs/2412.00440 |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866913592050188288 |
|---|---|
| author | Wang, Haicheng Ju, Chen Lin, Weixiong Xiao, Shuai Chen, Mengting Huang, Yixuan Liu, Chang Yao, Mingshuai Lan, Jinsong Chen, Ying Liu, Qingwen Wang, Yanfeng |
| author_facet | Wang, Haicheng Ju, Chen Lin, Weixiong Xiao, Shuai Chen, Mengting Huang, Yixuan Liu, Chang Yao, Mingshuai Lan, Jinsong Chen, Ying Liu, Qingwen Wang, Yanfeng |
| contents | In rapidly evolving field of vision-language models (VLMs), contrastive language-image pre-training (CLIP) has made significant strides, becoming foundation for various downstream tasks. However, relying on one-to-one (image, text) contrastive paradigm to learn alignment from large-scale messy web data, CLIP faces a serious myopic dilemma, resulting in biases towards monotonous short texts and shallow visual expressivity. To overcome these issues, this paper advances CLIP into one novel holistic paradigm, by updating both diverse data and alignment optimization. To obtain colorful data with low cost, we use image-to-text captioning to generate multi-texts for each image, from multiple perspectives, granularities, and hierarchies. Two gadgets are proposed to encourage textual diversity. To match such (image, multi-texts) pairs, we modify the CLIP image encoder into multi-branch, and propose multi-to-multi contrastive optimization for image-text part-to-part matching. As a result, diverse visual embeddings are learned for each image, bringing good interpretability and generalization. Extensive experiments and ablations across over ten benchmarks indicate that our holistic CLIP significantly outperforms existing myopic CLIP, including image-text retrieval, open-vocabulary classification, and dense visual tasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_00440 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Advancing Myopia To Holism: Fully Contrastive Language-Image Pre-training Wang, Haicheng Ju, Chen Lin, Weixiong Xiao, Shuai Chen, Mengting Huang, Yixuan Liu, Chang Yao, Mingshuai Lan, Jinsong Chen, Ying Liu, Qingwen Wang, Yanfeng Computer Vision and Pattern Recognition In rapidly evolving field of vision-language models (VLMs), contrastive language-image pre-training (CLIP) has made significant strides, becoming foundation for various downstream tasks. However, relying on one-to-one (image, text) contrastive paradigm to learn alignment from large-scale messy web data, CLIP faces a serious myopic dilemma, resulting in biases towards monotonous short texts and shallow visual expressivity. To overcome these issues, this paper advances CLIP into one novel holistic paradigm, by updating both diverse data and alignment optimization. To obtain colorful data with low cost, we use image-to-text captioning to generate multi-texts for each image, from multiple perspectives, granularities, and hierarchies. Two gadgets are proposed to encourage textual diversity. To match such (image, multi-texts) pairs, we modify the CLIP image encoder into multi-branch, and propose multi-to-multi contrastive optimization for image-text part-to-part matching. As a result, diverse visual embeddings are learned for each image, bringing good interpretability and generalization. Extensive experiments and ablations across over ten benchmarks indicate that our holistic CLIP significantly outperforms existing myopic CLIP, including image-text retrieval, open-vocabulary classification, and dense visual tasks. |
| title | Advancing Myopia To Holism: Fully Contrastive Language-Image Pre-training |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2412.00440 |