Retrieval-Augmented Natural Language Reasoning for Explainable Visual Question Answering
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866917763516203008 |
|---|---|
| author | Lim, Su Hyeon Kim, Minkuk Kim, Hyeon Bae Kim, Seong Tae |
| author_facet | Lim, Su Hyeon Kim, Minkuk Kim, Hyeon Bae Kim, Seong Tae |
| contents | Visual Question Answering with Natural Language Explanation (VQA-NLE) task is challenging due to its high demand for reasoning-based inference. Recent VQA-NLE studies focus on enhancing model networks to amplify the model's reasoning capability but this approach is resource-consuming and unstable. In this work, we introduce a new VQA-NLE model, ReRe (Retrieval-augmented natural language Reasoning), using leverage retrieval information from the memory to aid in generating accurate answers and persuasive explanations without relying on complex networks and extra datasets. ReRe is an encoder-decoder architecture model using a pre-trained clip vision encoder and a pre-trained GPT-2 language model as a decoder. Cross-attention layers are added in the GPT-2 for processing retrieval features. ReRe outperforms previous methods in VQA accuracy and explanation score and shows improvement in NLE with more persuasive, reliability. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_17006 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Retrieval-Augmented Natural Language Reasoning for Explainable Visual Question Answering Lim, Su Hyeon Kim, Minkuk Kim, Hyeon Bae Kim, Seong Tae Computer Vision and Pattern Recognition Visual Question Answering with Natural Language Explanation (VQA-NLE) task is challenging due to its high demand for reasoning-based inference. Recent VQA-NLE studies focus on enhancing model networks to amplify the model's reasoning capability but this approach is resource-consuming and unstable. In this work, we introduce a new VQA-NLE model, ReRe (Retrieval-augmented natural language Reasoning), using leverage retrieval information from the memory to aid in generating accurate answers and persuasive explanations without relying on complex networks and extra datasets. ReRe is an encoder-decoder architecture model using a pre-trained clip vision encoder and a pre-trained GPT-2 language model as a decoder. Cross-attention layers are added in the GPT-2 for processing retrieval features. ReRe outperforms previous methods in VQA accuracy and explanation score and shows improvement in NLE with more persuasive, reliability. |
| title | Retrieval-Augmented Natural Language Reasoning for Explainable Visual Question Answering |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2408.17006 |