Omni-Embed-Nemotron: A Unified Multimodal Retrieval Model for Text, Image, Audio, and Video
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866912626874777600 |
|---|---|
| author | Xu, Mengyao Zhou, Wenfei Babakhin, Yauhen Moreira, Gabriel Ak, Ronay Osmulski, Radek Liu, Bo Oldridge, Even Schifferer, Benedikt |
| author_facet | Xu, Mengyao Zhou, Wenfei Babakhin, Yauhen Moreira, Gabriel Ak, Ronay Osmulski, Radek Liu, Bo Oldridge, Even Schifferer, Benedikt |
| contents | We present Omni-Embed-Nemotron, a unified multimodal retrieval embedding model developed to handle the increasing complexity of real-world information needs. While Retrieval-Augmented Generation (RAG) has significantly advanced language models by incorporating external knowledge, existing text-based retrievers rely on clean, structured input and struggle with the visually and semantically rich content found in real-world documents such as PDFs, slides, or videos. Recent work such as ColPali has shown that preserving document layout using image-based representations can improve retrieval quality. Building on this, and inspired by the capabilities of recent multimodal models such as Qwen2.5-Omni, we extend retrieval beyond text and images to also support audio and video modalities. Omni-Embed-Nemotron enables both cross-modal (e.g., text - video) and joint-modal (e.g., text - video+audio) retrieval using a single model. We describe the architecture, training setup, and evaluation results of Omni-Embed-Nemotron, and demonstrate its effectiveness in text, image, and video retrieval. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_03458 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Omni-Embed-Nemotron: A Unified Multimodal Retrieval Model for Text, Image, Audio, and Video Xu, Mengyao Zhou, Wenfei Babakhin, Yauhen Moreira, Gabriel Ak, Ronay Osmulski, Radek Liu, Bo Oldridge, Even Schifferer, Benedikt Computation and Language We present Omni-Embed-Nemotron, a unified multimodal retrieval embedding model developed to handle the increasing complexity of real-world information needs. While Retrieval-Augmented Generation (RAG) has significantly advanced language models by incorporating external knowledge, existing text-based retrievers rely on clean, structured input and struggle with the visually and semantically rich content found in real-world documents such as PDFs, slides, or videos. Recent work such as ColPali has shown that preserving document layout using image-based representations can improve retrieval quality. Building on this, and inspired by the capabilities of recent multimodal models such as Qwen2.5-Omni, we extend retrieval beyond text and images to also support audio and video modalities. Omni-Embed-Nemotron enables both cross-modal (e.g., text - video) and joint-modal (e.g., text - video+audio) retrieval using a single model. We describe the architecture, training setup, and evaluation results of Omni-Embed-Nemotron, and demonstrate its effectiveness in text, image, and video retrieval. |
| title | Omni-Embed-Nemotron: A Unified Multimodal Retrieval Model for Text, Image, Audio, and Video |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2510.03458 |