Seeing Through the MiRAGE: Evaluating Multimodal Retrieval Augmented Generation
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911740306915328 |
|---|---|
| author | Martin, Alexander Walden, William Kriz, Reno Zhang, Dengjia Sanders, Kate Yang, Eugene Jin, Chihsheng Van Durme, Benjamin |
| author_facet | Martin, Alexander Walden, William Kriz, Reno Zhang, Dengjia Sanders, Kate Yang, Eugene Jin, Chihsheng Van Durme, Benjamin |
| contents | We introduce MiRAGE, an evaluation framework for retrieval-augmented generation (RAG) from multimodal sources. As audiovisual media becomes a prevalent source of information online, it is essential for RAG systems to integrate information from these sources into generation. However, existing evaluations for RAG are text-centric, limiting their applicability to multimodal settings. MiRAGE is a claim-centric approach to multimodal RAG evaluation, consisting of InfoF1, which assesses factuality and information coverage, and CiteF1, which assesses citation support and completeness. We show that, when applied by humans, MiRAGE strongly aligns with extrinsic judgments of output quality. We additionally introduce an automatic implementation of MiRAGE as well as multimodal variants of three prominent text-based RAG metrics -- ALCE, ARGUE, and RAGAS -- demonstrating the limitations of text-centric work and laying the groundwork for automatic evaluation. We release open-source implementations and outline evaluation methods for multimodal RAG. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_24870 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Seeing Through the MiRAGE: Evaluating Multimodal Retrieval Augmented Generation Martin, Alexander Walden, William Kriz, Reno Zhang, Dengjia Sanders, Kate Yang, Eugene Jin, Chihsheng Van Durme, Benjamin Computation and Language Computer Vision and Pattern Recognition Information Retrieval We introduce MiRAGE, an evaluation framework for retrieval-augmented generation (RAG) from multimodal sources. As audiovisual media becomes a prevalent source of information online, it is essential for RAG systems to integrate information from these sources into generation. However, existing evaluations for RAG are text-centric, limiting their applicability to multimodal settings. MiRAGE is a claim-centric approach to multimodal RAG evaluation, consisting of InfoF1, which assesses factuality and information coverage, and CiteF1, which assesses citation support and completeness. We show that, when applied by humans, MiRAGE strongly aligns with extrinsic judgments of output quality. We additionally introduce an automatic implementation of MiRAGE as well as multimodal variants of three prominent text-based RAG metrics -- ALCE, ARGUE, and RAGAS -- demonstrating the limitations of text-centric work and laying the groundwork for automatic evaluation. We release open-source implementations and outline evaluation methods for multimodal RAG. |
| title | Seeing Through the MiRAGE: Evaluating Multimodal Retrieval Augmented Generation |
| topic | Computation and Language Computer Vision and Pattern Recognition Information Retrieval |
| url | https://arxiv.org/abs/2510.24870 |