Question Aware Vision Transformer for Multimodal Reasoning
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914671172255744 |
|---|---|
| author | Ganz, Roy Kittenplon, Yair Aberdam, Aviad Avraham, Elad Ben Nuriel, Oren Mazor, Shai Litman, Ron |
| author_facet | Ganz, Roy Kittenplon, Yair Aberdam, Aviad Avraham, Elad Ben Nuriel, Oren Mazor, Shai Litman, Ron |
| contents | Vision-Language (VL) models have gained significant research focus, enabling remarkable advances in multimodal reasoning. These architectures typically comprise a vision encoder, a Large Language Model (LLM), and a projection module that aligns visual features with the LLM's representation space. Despite their success, a critical limitation persists: the vision encoding process remains decoupled from user queries, often in the form of image-related questions. Consequently, the resulting visual features may not be optimally attuned to the query-specific elements of the image. To address this, we introduce QA-ViT, a Question Aware Vision Transformer approach for multimodal reasoning, which embeds question awareness directly within the vision encoder. This integration results in dynamic visual features focusing on relevant image aspects to the posed question. QA-ViT is model-agnostic and can be incorporated efficiently into any VL architecture. Extensive experiments demonstrate the effectiveness of applying our method to various multimodal architectures, leading to consistent improvement across diverse tasks and showcasing its potential for enhancing visual and scene-text understanding. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_05472 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Question Aware Vision Transformer for Multimodal Reasoning Ganz, Roy Kittenplon, Yair Aberdam, Aviad Avraham, Elad Ben Nuriel, Oren Mazor, Shai Litman, Ron Computer Vision and Pattern Recognition Vision-Language (VL) models have gained significant research focus, enabling remarkable advances in multimodal reasoning. These architectures typically comprise a vision encoder, a Large Language Model (LLM), and a projection module that aligns visual features with the LLM's representation space. Despite their success, a critical limitation persists: the vision encoding process remains decoupled from user queries, often in the form of image-related questions. Consequently, the resulting visual features may not be optimally attuned to the query-specific elements of the image. To address this, we introduce QA-ViT, a Question Aware Vision Transformer approach for multimodal reasoning, which embeds question awareness directly within the vision encoder. This integration results in dynamic visual features focusing on relevant image aspects to the posed question. QA-ViT is model-agnostic and can be incorporated efficiently into any VL architecture. Extensive experiments demonstrate the effectiveness of applying our method to various multimodal architectures, leading to consistent improvement across diverse tasks and showcasing its potential for enhancing visual and scene-text understanding. |
| title | Question Aware Vision Transformer for Multimodal Reasoning |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2402.05472 |