Retrieval-Augmented Visual Question Answering via Built-in Autoregressive Search Engines
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866910841191792640 |
|---|---|
| author | Long, Xinwei Ma, Zhiyuan Hua, Ermo Zhang, Kaiyan Qi, Biqing Zhou, Bowen |
| author_facet | Long, Xinwei Ma, Zhiyuan Hua, Ermo Zhang, Kaiyan Qi, Biqing Zhou, Bowen |
| contents | Retrieval-augmented generation (RAG) has emerged to address the knowledge-intensive visual question answering (VQA) task. Current methods mainly employ separate retrieval and generation modules to acquire external knowledge and generate answers, respectively. We propose ReAuSE, an alternative to the previous RAG model for the knowledge-based VQA task, which seamlessly integrates knowledge retriever into the generative multi-modal large language model, serving as a built-in search engine. Specifically, our model functions both as a generative retriever and an accurate answer generator. It not only helps retrieve documents from the knowledge base by producing identifiers for each document, but it also answers visual questions based on the retrieved documents. Furthermore, we propose a reinforced retrieval calibration module from relevance feedback to improve retrieval performance and align with the preferences for accurate answer generation. Extensive experiments on two representative OKVQA and A-OKVQA datasets demonstrate significant improvements ranging from 2.9\% to 9.6\% across all evaluation metrics when compared to strong baselines. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_16641 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Retrieval-Augmented Visual Question Answering via Built-in Autoregressive Search Engines Long, Xinwei Ma, Zhiyuan Hua, Ermo Zhang, Kaiyan Qi, Biqing Zhou, Bowen Computer Vision and Pattern Recognition Computation and Language Information Retrieval Retrieval-augmented generation (RAG) has emerged to address the knowledge-intensive visual question answering (VQA) task. Current methods mainly employ separate retrieval and generation modules to acquire external knowledge and generate answers, respectively. We propose ReAuSE, an alternative to the previous RAG model for the knowledge-based VQA task, which seamlessly integrates knowledge retriever into the generative multi-modal large language model, serving as a built-in search engine. Specifically, our model functions both as a generative retriever and an accurate answer generator. It not only helps retrieve documents from the knowledge base by producing identifiers for each document, but it also answers visual questions based on the retrieved documents. Furthermore, we propose a reinforced retrieval calibration module from relevance feedback to improve retrieval performance and align with the preferences for accurate answer generation. Extensive experiments on two representative OKVQA and A-OKVQA datasets demonstrate significant improvements ranging from 2.9\% to 9.6\% across all evaluation metrics when compared to strong baselines. |
| title | Retrieval-Augmented Visual Question Answering via Built-in Autoregressive Search Engines |
| topic | Computer Vision and Pattern Recognition Computation and Language Information Retrieval |
| url | https://arxiv.org/abs/2502.16641 |