jina-clip-v2: Multilingual Multimodal Embeddings for Text and Images
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866908336727785472 |
|---|---|
| author | Koukounas, Andreas Mastrapas, Georgios Eslami, Sedigheh Wang, Bo Akram, Mohammad Kalim Günther, Michael Mohr, Isabelle Sturua, Saba Wang, Nan Xiao, Han |
| author_facet | Koukounas, Andreas Mastrapas, Georgios Eslami, Sedigheh Wang, Bo Akram, Mohammad Kalim Günther, Michael Mohr, Isabelle Sturua, Saba Wang, Nan Xiao, Han |
| contents | Contrastive Language-Image Pretraining (CLIP) has been widely used for crossmodal information retrieval and multimodal understanding tasks. However, CLIP models are mainly optimized for crossmodal vision-language tasks and underperform in single-mode text tasks. Moreover, these models are often trained on English datasets and therefore lack multilingual understanding. Additionally, from a visual understanding perspective, previous CLIP-based models exhibit insufficient understanding of visually rich documents. In this work, we propose jina-clip-v2, a contrastive vision-language model trained on text pairs, triplets and image-text pairs via a multi-task and multi-stage contrastive learning paradigm in order to support both text-only and crossmodal tasks. We employ a multilingual text encoder and expand the training dataset to include multilingual texts from 29 non-English languages, including Hindi, Chinese, German, French, and others, as well as images of visually rich documents. We evaluate the model's performance and show that jina-clip-v2 achieves notable improvements over state-of-the-art CLIP-based models in zero-shot text-only retrieval, semantic textual similarity, and crossmodal retrieval tasks in both English and multilingual settings. jina-clip-v2 also provides for flexibility in embedding dimensionality, enabling users to select the granularity of the representations. jina-clip-v2 is publicly available at https://huggingface.co/jinaai/jina-clip-v2. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_08802 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | jina-clip-v2: Multilingual Multimodal Embeddings for Text and Images Koukounas, Andreas Mastrapas, Georgios Eslami, Sedigheh Wang, Bo Akram, Mohammad Kalim Günther, Michael Mohr, Isabelle Sturua, Saba Wang, Nan Xiao, Han Computation and Language Computer Vision and Pattern Recognition Information Retrieval 68T50 I.2.7; I.2.10 Contrastive Language-Image Pretraining (CLIP) has been widely used for crossmodal information retrieval and multimodal understanding tasks. However, CLIP models are mainly optimized for crossmodal vision-language tasks and underperform in single-mode text tasks. Moreover, these models are often trained on English datasets and therefore lack multilingual understanding. Additionally, from a visual understanding perspective, previous CLIP-based models exhibit insufficient understanding of visually rich documents. In this work, we propose jina-clip-v2, a contrastive vision-language model trained on text pairs, triplets and image-text pairs via a multi-task and multi-stage contrastive learning paradigm in order to support both text-only and crossmodal tasks. We employ a multilingual text encoder and expand the training dataset to include multilingual texts from 29 non-English languages, including Hindi, Chinese, German, French, and others, as well as images of visually rich documents. We evaluate the model's performance and show that jina-clip-v2 achieves notable improvements over state-of-the-art CLIP-based models in zero-shot text-only retrieval, semantic textual similarity, and crossmodal retrieval tasks in both English and multilingual settings. jina-clip-v2 also provides for flexibility in embedding dimensionality, enabling users to select the granularity of the representations. jina-clip-v2 is publicly available at https://huggingface.co/jinaai/jina-clip-v2. |
| title | jina-clip-v2: Multilingual Multimodal Embeddings for Text and Images |
| topic | Computation and Language Computer Vision and Pattern Recognition Information Retrieval 68T50 I.2.7; I.2.10 |
| url | https://arxiv.org/abs/2412.08802 |