Entity Image and Mixed-Modal Image Retrieval Datasets
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912410900627456 |
|---|---|
| author | Blaga, Cristian-Ioan Suganthan, Paul Dua, Sahil Srinivasan, Krishna Alfonseca, Enrique Dornbach, Peter Duerig, Tom Zitouni, Imed Dong, Zhe |
| author_facet | Blaga, Cristian-Ioan Suganthan, Paul Dua, Sahil Srinivasan, Krishna Alfonseca, Enrique Dornbach, Peter Duerig, Tom Zitouni, Imed Dong, Zhe |
| contents | Despite advances in multimodal learning, challenging benchmarks for mixed-modal image retrieval that combines visual and textual information are lacking. This paper introduces a novel benchmark to rigorously evaluate image retrieval that demands deep cross-modal contextual understanding. We present two new datasets: the Entity Image Dataset (EI), providing canonical images for Wikipedia entities, and the Mixed-Modal Image Retrieval Dataset (MMIR), derived from the WIT dataset. The MMIR benchmark features two challenging query types requiring models to ground textual descriptions in the context of provided visual entities: single entity-image queries (one entity image with descriptive text) and multi-entity-image queries (multiple entity images with relational text). We empirically validate the benchmark's utility as both a training corpus and an evaluation set for mixed-modal retrieval. The quality of both datasets is further affirmed through crowd-sourced human annotations. The datasets are accessible through the GitHub page: https://github.com/google-research-datasets/wit-retrieval. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_02291 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Entity Image and Mixed-Modal Image Retrieval Datasets Blaga, Cristian-Ioan Suganthan, Paul Dua, Sahil Srinivasan, Krishna Alfonseca, Enrique Dornbach, Peter Duerig, Tom Zitouni, Imed Dong, Zhe Computer Vision and Pattern Recognition Information Retrieval Despite advances in multimodal learning, challenging benchmarks for mixed-modal image retrieval that combines visual and textual information are lacking. This paper introduces a novel benchmark to rigorously evaluate image retrieval that demands deep cross-modal contextual understanding. We present two new datasets: the Entity Image Dataset (EI), providing canonical images for Wikipedia entities, and the Mixed-Modal Image Retrieval Dataset (MMIR), derived from the WIT dataset. The MMIR benchmark features two challenging query types requiring models to ground textual descriptions in the context of provided visual entities: single entity-image queries (one entity image with descriptive text) and multi-entity-image queries (multiple entity images with relational text). We empirically validate the benchmark's utility as both a training corpus and an evaluation set for mixed-modal retrieval. The quality of both datasets is further affirmed through crowd-sourced human annotations. The datasets are accessible through the GitHub page: https://github.com/google-research-datasets/wit-retrieval. |
| title | Entity Image and Mixed-Modal Image Retrieval Datasets |
| topic | Computer Vision and Pattern Recognition Information Retrieval |
| url | https://arxiv.org/abs/2506.02291 |