LLM2CLIP: Powerful Language Model Unlocks Richer Cross-Modality Representation
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866918354545016832 |
|---|---|
| author | Huang, Weiquan Wu, Aoqi Yang, Yifan Luo, Xufang Yang, Yuqing Naseem, Usman Wang, Chunyu Wang, Chunyu Dai, Qi Dai, Xiyang Chen, Dongdong Luo, Chong Qiu, Lili Hu, Liang |
| author_facet | Huang, Weiquan Wu, Aoqi Yang, Yifan Luo, Xufang Yang, Yuqing Naseem, Usman Wang, Chunyu Wang, Chunyu Dai, Qi Dai, Xiyang Chen, Dongdong Luo, Chong Qiu, Lili Hu, Liang |
| contents | CLIP is a seminal multimodal model that maps images and text into a shared representation space through contrastive learning on billions of image-caption pairs. Inspired by the rapid progress of large language models (LLMs), we investigate how the superior linguistic understanding and broad world knowledge of LLMs can further strengthen CLIP, particularly in handling long and complex captions. We introduce an efficient fine-tuning framework that embeds an LLM into a pretrained CLIP while incurring nearly the same training cost as standard CLIP fine-tuning. Our method first converts the LLM into an embedding-compatible form for the CLIP setting, and then couples it with the pretrained CLIP vision encoder through a lightweight adaptor trained on only a few million image-caption pairs. With this strategy, we achieve large performance gains without large-scale retraining, outperforming state-of-the-art CLIP variants such as EVA02 and SigLIP-2. The LLM-enhanced CLIP delivers consistent improvements across a wide range of downstream tasks, including linear-probe classification, zero-shot image-text retrieval with both short and long captions (in English and other languages), zero-shot and supervised image segmentation, object detection, and serving as a tokenizer backbone for multimodal large-model benchmarks. Code and models are available at: https://aka.ms/llm2clip |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_04997 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | LLM2CLIP: Powerful Language Model Unlocks Richer Cross-Modality Representation Huang, Weiquan Wu, Aoqi Yang, Yifan Luo, Xufang Yang, Yuqing Naseem, Usman Wang, Chunyu Wang, Chunyu Dai, Qi Dai, Xiyang Chen, Dongdong Luo, Chong Qiu, Lili Hu, Liang Computer Vision and Pattern Recognition Computation and Language CLIP is a seminal multimodal model that maps images and text into a shared representation space through contrastive learning on billions of image-caption pairs. Inspired by the rapid progress of large language models (LLMs), we investigate how the superior linguistic understanding and broad world knowledge of LLMs can further strengthen CLIP, particularly in handling long and complex captions. We introduce an efficient fine-tuning framework that embeds an LLM into a pretrained CLIP while incurring nearly the same training cost as standard CLIP fine-tuning. Our method first converts the LLM into an embedding-compatible form for the CLIP setting, and then couples it with the pretrained CLIP vision encoder through a lightweight adaptor trained on only a few million image-caption pairs. With this strategy, we achieve large performance gains without large-scale retraining, outperforming state-of-the-art CLIP variants such as EVA02 and SigLIP-2. The LLM-enhanced CLIP delivers consistent improvements across a wide range of downstream tasks, including linear-probe classification, zero-shot image-text retrieval with both short and long captions (in English and other languages), zero-shot and supervised image segmentation, object detection, and serving as a tokenizer backbone for multimodal large-model benchmarks. Code and models are available at: https://aka.ms/llm2clip |
| title | LLM2CLIP: Powerful Language Model Unlocks Richer Cross-Modality Representation |
| topic | Computer Vision and Pattern Recognition Computation and Language |
| url | https://arxiv.org/abs/2411.04997 |