Integrating Query-aware Segmentation and Cross-Attention for Robust VQA
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866911957799403520 |
|---|---|
| author | Choi, Wonjun Lee, Sangbeom Lee, Seungyeon Jung, Heechul Lee, Dong-Gyu |
| author_facet | Choi, Wonjun Lee, Sangbeom Lee, Seungyeon Jung, Heechul Lee, Dong-Gyu |
| contents | This paper introduces a method for VizWiz-VQA using LVLM with trainable cross-attention and LoRA finetuning. We train the model with the following conditions: 1) Training with original images. 2) Training with enhanced images using CLIPSeg to highlight or contrast the original image. 3) Training with integrating the output features of Vision Transformer (ViT) and CLIPSeg features of the original images. Then, we ensemble the results based on Levenshtein distance to enhance the prediction of the final answer. In the experiments, we demonstrate and analyze the proposed method's effectiveness. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_12055 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Integrating Query-aware Segmentation and Cross-Attention for Robust VQA Choi, Wonjun Lee, Sangbeom Lee, Seungyeon Jung, Heechul Lee, Dong-Gyu Computer Vision and Pattern Recognition This paper introduces a method for VizWiz-VQA using LVLM with trainable cross-attention and LoRA finetuning. We train the model with the following conditions: 1) Training with original images. 2) Training with enhanced images using CLIPSeg to highlight or contrast the original image. 3) Training with integrating the output features of Vision Transformer (ViT) and CLIPSeg features of the original images. Then, we ensemble the results based on Levenshtein distance to enhance the prediction of the final answer. In the experiments, we demonstrate and analyze the proposed method's effectiveness. |
| title | Integrating Query-aware Segmentation and Cross-Attention for Robust VQA |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2407.12055 |