Indexing Multimodal Language Models for Large-scale Image Retrieval
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2026
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866915937878278144 |
|---|---|
| author | Tharwat, Bahey Kordopatis-Zilos, Giorgos Suma, Pavel Reid, Ian Tolias, Giorgos |
| author_facet | Tharwat, Bahey Kordopatis-Zilos, Giorgos Suma, Pavel Reid, Ian Tolias, Giorgos |
| contents | Multimodal Large Language Models (MLLMs) have demonstrated strong cross-modal reasoning capabilities, yet their potential for vision-only tasks remains underexplored. We investigate MLLMs as training-free similarity estimators for instance-level image-to-image retrieval. Our approach prompts the model with paired images and converts next-token probabilities into similarity scores, enabling zero-shot re-ranking within large-scale retrieval pipelines. This design avoids specialized architectures and fine-tuning, leveraging the rich visual discrimination learned during multimodal pre-training. We address scalability by combining MLLMs with memory-efficient indexing and top-$k$ candidate re-ranking. Experiments across diverse benchmarks show that MLLMs outperform task-specific re-rankers outside their native domains and exhibit superior robustness to clutter, occlusion, and small objects. Despite strong results, we identify failure modes under severe appearance changes, highlighting opportunities for future research. Our findings position MLLMs as a promising alternative for open-world large-scale image retrieval. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_13268 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Indexing Multimodal Language Models for Large-scale Image Retrieval Tharwat, Bahey Kordopatis-Zilos, Giorgos Suma, Pavel Reid, Ian Tolias, Giorgos Computer Vision and Pattern Recognition Computation and Language Information Retrieval Multimodal Large Language Models (MLLMs) have demonstrated strong cross-modal reasoning capabilities, yet their potential for vision-only tasks remains underexplored. We investigate MLLMs as training-free similarity estimators for instance-level image-to-image retrieval. Our approach prompts the model with paired images and converts next-token probabilities into similarity scores, enabling zero-shot re-ranking within large-scale retrieval pipelines. This design avoids specialized architectures and fine-tuning, leveraging the rich visual discrimination learned during multimodal pre-training. We address scalability by combining MLLMs with memory-efficient indexing and top-$k$ candidate re-ranking. Experiments across diverse benchmarks show that MLLMs outperform task-specific re-rankers outside their native domains and exhibit superior robustness to clutter, occlusion, and small objects. Despite strong results, we identify failure modes under severe appearance changes, highlighting opportunities for future research. Our findings position MLLMs as a promising alternative for open-world large-scale image retrieval. |
| title | Indexing Multimodal Language Models for Large-scale Image Retrieval |
| topic | Computer Vision and Pattern Recognition Computation and Language Information Retrieval |
| url | https://arxiv.org/abs/2604.13268 |