Ask in Any Modality: A Comprehensive Survey on Multimodal Retrieval-Augmented Generation
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866908388994056192 |
|---|---|
| author | Abootorabi, Mohammad Mahdi Zobeiri, Amirhosein Dehghani, Mahdi Mohammadkhani, Mohammadali Mohammadi, Bardia Ghahroodi, Omid Baghshah, Mahdieh Soleymani Asgari, Ehsaneddin |
| author_facet | Abootorabi, Mohammad Mahdi Zobeiri, Amirhosein Dehghani, Mahdi Mohammadkhani, Mohammadali Mohammadi, Bardia Ghahroodi, Omid Baghshah, Mahdieh Soleymani Asgari, Ehsaneddin |
| contents | Large Language Models (LLMs) suffer from hallucinations and outdated knowledge due to their reliance on static training data. Retrieval-Augmented Generation (RAG) mitigates these issues by integrating external dynamic information for improved factual grounding. With advances in multimodal learning, Multimodal RAG extends this approach by incorporating multiple modalities such as text, images, audio, and video to enhance the generated outputs. However, cross-modal alignment and reasoning introduce unique challenges beyond those in unimodal RAG. This survey offers a structured and comprehensive analysis of Multimodal RAG systems, covering datasets, benchmarks, metrics, evaluation, methodologies, and innovations in retrieval, fusion, augmentation, and generation. We review training strategies, robustness enhancements, loss functions, and agent-based approaches, while also exploring the diverse Multimodal RAG scenarios. In addition, we outline open challenges and future directions to guide research in this evolving field. This survey lays the foundation for developing more capable and reliable AI systems that effectively leverage multimodal dynamic external knowledge bases. All resources are publicly available at https://github.com/llm-lab-org/Multimodal-RAG-Survey. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_08826 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Ask in Any Modality: A Comprehensive Survey on Multimodal Retrieval-Augmented Generation Abootorabi, Mohammad Mahdi Zobeiri, Amirhosein Dehghani, Mahdi Mohammadkhani, Mohammadali Mohammadi, Bardia Ghahroodi, Omid Baghshah, Mahdieh Soleymani Asgari, Ehsaneddin Computation and Language Artificial Intelligence Information Retrieval Large Language Models (LLMs) suffer from hallucinations and outdated knowledge due to their reliance on static training data. Retrieval-Augmented Generation (RAG) mitigates these issues by integrating external dynamic information for improved factual grounding. With advances in multimodal learning, Multimodal RAG extends this approach by incorporating multiple modalities such as text, images, audio, and video to enhance the generated outputs. However, cross-modal alignment and reasoning introduce unique challenges beyond those in unimodal RAG. This survey offers a structured and comprehensive analysis of Multimodal RAG systems, covering datasets, benchmarks, metrics, evaluation, methodologies, and innovations in retrieval, fusion, augmentation, and generation. We review training strategies, robustness enhancements, loss functions, and agent-based approaches, while also exploring the diverse Multimodal RAG scenarios. In addition, we outline open challenges and future directions to guide research in this evolving field. This survey lays the foundation for developing more capable and reliable AI systems that effectively leverage multimodal dynamic external knowledge bases. All resources are publicly available at https://github.com/llm-lab-org/Multimodal-RAG-Survey. |
| title | Ask in Any Modality: A Comprehensive Survey on Multimodal Retrieval-Augmented Generation |
| topic | Computation and Language Artificial Intelligence Information Retrieval |
| url | https://arxiv.org/abs/2502.08826 |