VLM2Vec-V2: Advancing Multimodal Embedding for Videos, Images, and Visual Documents
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866913929617211392 |
|---|---|
| author | Meng, Rui Jiang, Ziyan Liu, Ye Su, Mingyi Yang, Xinyi Fu, Yuepeng Qin, Can Chen, Zeyuan Xu, Ran Xiong, Caiming Zhou, Yingbo Chen, Wenhu Yavuz, Semih |
| author_facet | Meng, Rui Jiang, Ziyan Liu, Ye Su, Mingyi Yang, Xinyi Fu, Yuepeng Qin, Can Chen, Zeyuan Xu, Ran Xiong, Caiming Zhou, Yingbo Chen, Wenhu Yavuz, Semih |
| contents | Multimodal embedding models have been crucial in enabling various downstream tasks such as semantic similarity, information retrieval, and clustering over different modalities. However, existing multimodal embeddings like VLM2Vec, E5-V, GME are predominantly focused on natural images, with limited support for other visual forms such as videos and visual documents. This restricts their applicability in real-world scenarios, including AI agents, multi-modal search and recommendation, and retrieval-augmented generation (RAG). To close this gap, we propose VLM2Vec-V2, a unified framework for learning embeddings across diverse visual forms. First, we introduce MMEB-V2, a comprehensive benchmark that extends MMEB with five new task types: visual document retrieval, video retrieval, temporal grounding, video classification and video question answering - spanning text, image, video, and visual document inputs. Next, we train VLM2Vec-V2, a general-purpose embedding model that supports text, image, video, and visual document inputs. Extensive experiments show that VLM2Vec-V2 achieves strong performance not only on the newly introduced video and document retrieval tasks, but also improves over prior baselines on the original image benchmarks. Through extensive evaluation, our study offers insights into the generalizability of various multimodal embedding models and highlights effective strategies for unified embedding learning, laying the groundwork for more scalable and adaptable representation learning in both research and real-world settings. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_04590 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | VLM2Vec-V2: Advancing Multimodal Embedding for Videos, Images, and Visual Documents Meng, Rui Jiang, Ziyan Liu, Ye Su, Mingyi Yang, Xinyi Fu, Yuepeng Qin, Can Chen, Zeyuan Xu, Ran Xiong, Caiming Zhou, Yingbo Chen, Wenhu Yavuz, Semih Computer Vision and Pattern Recognition Computation and Language Multimodal embedding models have been crucial in enabling various downstream tasks such as semantic similarity, information retrieval, and clustering over different modalities. However, existing multimodal embeddings like VLM2Vec, E5-V, GME are predominantly focused on natural images, with limited support for other visual forms such as videos and visual documents. This restricts their applicability in real-world scenarios, including AI agents, multi-modal search and recommendation, and retrieval-augmented generation (RAG). To close this gap, we propose VLM2Vec-V2, a unified framework for learning embeddings across diverse visual forms. First, we introduce MMEB-V2, a comprehensive benchmark that extends MMEB with five new task types: visual document retrieval, video retrieval, temporal grounding, video classification and video question answering - spanning text, image, video, and visual document inputs. Next, we train VLM2Vec-V2, a general-purpose embedding model that supports text, image, video, and visual document inputs. Extensive experiments show that VLM2Vec-V2 achieves strong performance not only on the newly introduced video and document retrieval tasks, but also improves over prior baselines on the original image benchmarks. Through extensive evaluation, our study offers insights into the generalizability of various multimodal embedding models and highlights effective strategies for unified embedding learning, laying the groundwork for more scalable and adaptable representation learning in both research and real-world settings. |
| title | VLM2Vec-V2: Advancing Multimodal Embedding for Videos, Images, and Visual Documents |
| topic | Computer Vision and Pattern Recognition Computation and Language |
| url | https://arxiv.org/abs/2507.04590 |