Wiki-LLaVA: Hierarchical Retrieval-Augmented Generation for Multimodal LLMs
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866916257062715392 |
|---|---|
| author | Caffagni, Davide Cocchi, Federico Moratelli, Nicholas Sarto, Sara Cornia, Marcella Baraldi, Lorenzo Cucchiara, Rita |
| author_facet | Caffagni, Davide Cocchi, Federico Moratelli, Nicholas Sarto, Sara Cornia, Marcella Baraldi, Lorenzo Cucchiara, Rita |
| contents | Multimodal LLMs are the natural evolution of LLMs, and enlarge their capabilities so as to work beyond the pure textual modality. As research is being carried out to design novel architectures and vision-and-language adapters, in this paper we concentrate on endowing such models with the capability of answering questions that require external knowledge. Our approach, termed Wiki-LLaVA, aims at integrating an external knowledge source of multimodal documents, which is accessed through a hierarchical retrieval pipeline. Relevant passages, using this approach, are retrieved from the external knowledge source and employed as additional context for the LLM, augmenting the effectiveness and precision of generated dialogues. We conduct extensive experiments on datasets tailored for visual question answering with external data and demonstrate the appropriateness of our approach. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_15406 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Wiki-LLaVA: Hierarchical Retrieval-Augmented Generation for Multimodal LLMs Caffagni, Davide Cocchi, Federico Moratelli, Nicholas Sarto, Sara Cornia, Marcella Baraldi, Lorenzo Cucchiara, Rita Computer Vision and Pattern Recognition Artificial Intelligence Computation and Language Multimedia Multimodal LLMs are the natural evolution of LLMs, and enlarge their capabilities so as to work beyond the pure textual modality. As research is being carried out to design novel architectures and vision-and-language adapters, in this paper we concentrate on endowing such models with the capability of answering questions that require external knowledge. Our approach, termed Wiki-LLaVA, aims at integrating an external knowledge source of multimodal documents, which is accessed through a hierarchical retrieval pipeline. Relevant passages, using this approach, are retrieved from the external knowledge source and employed as additional context for the LLM, augmenting the effectiveness and precision of generated dialogues. We conduct extensive experiments on datasets tailored for visual question answering with external data and demonstrate the appropriateness of our approach. |
| title | Wiki-LLaVA: Hierarchical Retrieval-Augmented Generation for Multimodal LLMs |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Computation and Language Multimedia |
| url | https://arxiv.org/abs/2404.15406 |