Harnessing Frozen Unimodal Encoders for Flexible Multimodal Alignment
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866915210463281152 |
|---|---|
| author | Maniparambil, Mayug Akshulakov, Raiymbek Djilali, Yasser Abdelaziz Dahou Narayan, Sanath Singh, Ankit O'Connor, Noel E. |
| author_facet | Maniparambil, Mayug Akshulakov, Raiymbek Djilali, Yasser Abdelaziz Dahou Narayan, Sanath Singh, Ankit O'Connor, Noel E. |
| contents | Recent contrastive multimodal vision-language models like CLIP have demonstrated robust open-world semantic understanding, becoming the standard image backbones for vision-language applications. However, recent findings suggest high semantic similarity between well-trained unimodal encoders, which raises a key question: Is there a plausible way to connect unimodal backbones for vision-language tasks? To this end, we propose a novel framework that aligns vision and language using frozen unimodal encoders. It involves selecting semantically similar encoders in the latent space, curating a concept-rich dataset of image-caption pairs, and training simple MLP projectors. We evaluated our approach on 12 zero-shot classification datasets and 2 image-text retrieval datasets. Our best model, utilizing DINOv2 and All-Roberta-Large text encoder, achieves 76\(\%\) accuracy on ImageNet with a 20-fold reduction in data and 65-fold reduction in compute requirements compared multi-modal alignment where models are trained from scratch. The proposed framework enhances the accessibility of multimodal model development while enabling flexible adaptation across diverse scenarios. Code and curated datasets are available at \texttt{github.com/mayug/freeze-align}. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_19425 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Harnessing Frozen Unimodal Encoders for Flexible Multimodal Alignment Maniparambil, Mayug Akshulakov, Raiymbek Djilali, Yasser Abdelaziz Dahou Narayan, Sanath Singh, Ankit O'Connor, Noel E. Computer Vision and Pattern Recognition Recent contrastive multimodal vision-language models like CLIP have demonstrated robust open-world semantic understanding, becoming the standard image backbones for vision-language applications. However, recent findings suggest high semantic similarity between well-trained unimodal encoders, which raises a key question: Is there a plausible way to connect unimodal backbones for vision-language tasks? To this end, we propose a novel framework that aligns vision and language using frozen unimodal encoders. It involves selecting semantically similar encoders in the latent space, curating a concept-rich dataset of image-caption pairs, and training simple MLP projectors. We evaluated our approach on 12 zero-shot classification datasets and 2 image-text retrieval datasets. Our best model, utilizing DINOv2 and All-Roberta-Large text encoder, achieves 76\(\%\) accuracy on ImageNet with a 20-fold reduction in data and 65-fold reduction in compute requirements compared multi-modal alignment where models are trained from scratch. The proposed framework enhances the accessibility of multimodal model development while enabling flexible adaptation across diverse scenarios. Code and curated datasets are available at \texttt{github.com/mayug/freeze-align}. |
| title | Harnessing Frozen Unimodal Encoders for Flexible Multimodal Alignment |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2409.19425 |