ColMate: Contrastive Late Interaction and Masked Text for Multimodal Document Retrieval
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866912683312283648 |
|---|---|
| author | Masry, Ahmed Thakkar, Megh Bechard, Patrice Madhusudhan, Sathwik Tejaswi Awal, Rabiul Mishra, Shambhavi Suresh, Akshay Kalkunte Daruru, Srivatsava Hoque, Enamul Gella, Spandana Scholak, Torsten Rajeswar, Sai |
| author_facet | Masry, Ahmed Thakkar, Megh Bechard, Patrice Madhusudhan, Sathwik Tejaswi Awal, Rabiul Mishra, Shambhavi Suresh, Akshay Kalkunte Daruru, Srivatsava Hoque, Enamul Gella, Spandana Scholak, Torsten Rajeswar, Sai |
| contents | Retrieval-augmented generation has proven practical when models require specialized knowledge or access to the latest data. However, existing methods for multimodal document retrieval often replicate techniques developed for text-only retrieval, whether in how they encode documents, define training objectives, or compute similarity scores. To address these limitations, we present ColMate, a document retrieval model that bridges the gap between multimodal representation learning and document retrieval. ColMate utilizes a novel OCR-based pretraining objective, a self-supervised masked contrastive learning objective, and a late interaction scoring mechanism more relevant to multimodal document structures and visual characteristics. ColMate obtains 3.61% improvements over existing retrieval models on the ViDoRe V2 benchmark, demonstrating stronger generalization to out-of-domain benchmarks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_00903 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | ColMate: Contrastive Late Interaction and Masked Text for Multimodal Document Retrieval Masry, Ahmed Thakkar, Megh Bechard, Patrice Madhusudhan, Sathwik Tejaswi Awal, Rabiul Mishra, Shambhavi Suresh, Akshay Kalkunte Daruru, Srivatsava Hoque, Enamul Gella, Spandana Scholak, Torsten Rajeswar, Sai Computation and Language Retrieval-augmented generation has proven practical when models require specialized knowledge or access to the latest data. However, existing methods for multimodal document retrieval often replicate techniques developed for text-only retrieval, whether in how they encode documents, define training objectives, or compute similarity scores. To address these limitations, we present ColMate, a document retrieval model that bridges the gap between multimodal representation learning and document retrieval. ColMate utilizes a novel OCR-based pretraining objective, a self-supervised masked contrastive learning objective, and a late interaction scoring mechanism more relevant to multimodal document structures and visual characteristics. ColMate obtains 3.61% improvements over existing retrieval models on the ViDoRe V2 benchmark, demonstrating stronger generalization to out-of-domain benchmarks. |
| title | ColMate: Contrastive Late Interaction and Masked Text for Multimodal Document Retrieval |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2511.00903 |