Mitigating Hallucination in Visual-Language Models via Re-Balancing Contrastive Decoding
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866913495877943296 |
|---|---|
| author | Liang, Xiaoyu Yu, Jiayuan Mu, Lianrui Zhuang, Jiedong Hu, Jiaqi Yang, Yuchen Ye, Jiangnan Lu, Lu Chen, Jian Hu, Haoji |
| author_facet | Liang, Xiaoyu Yu, Jiayuan Mu, Lianrui Zhuang, Jiedong Hu, Jiaqi Yang, Yuchen Ye, Jiangnan Lu, Lu Chen, Jian Hu, Haoji |
| contents | Although Visual-Language Models (VLMs) have shown impressive capabilities in tasks like visual question answering and image captioning, they still struggle with hallucinations. Analysis of attention distribution in these models shows that VLMs tend to processing textual tokens rather than visual tokens. This imbalance of attention distribution causes VLMs to favor textual knowledge in the case of multimodal knowledge conflicts, resulting in differences from the image information. In this paper, we propose Re-Balancing Contrastive Decoding (RBD) method, which employs textual and visual branches to recalibrate attention distribution in VLMs. Specifically, the textual branch injects image noise to stimulate the model's dependency on text, thereby reducing textual bias. Concurrently, the visual branch focuses on the selection of significant tokens, refining the attention mechanism to highlight the primary subject. This dual-branch strategy enables the RBD method to diminish textual bias while enhancing visual information. Experimental results demonstrate that our method, RBD, outperforms the existing methods by the CHAIR and POPE metrics, mitigate hallucinations without reducing the model's general capabilities. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_06485 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Mitigating Hallucination in Visual-Language Models via Re-Balancing Contrastive Decoding Liang, Xiaoyu Yu, Jiayuan Mu, Lianrui Zhuang, Jiedong Hu, Jiaqi Yang, Yuchen Ye, Jiangnan Lu, Lu Chen, Jian Hu, Haoji Computer Vision and Pattern Recognition Although Visual-Language Models (VLMs) have shown impressive capabilities in tasks like visual question answering and image captioning, they still struggle with hallucinations. Analysis of attention distribution in these models shows that VLMs tend to processing textual tokens rather than visual tokens. This imbalance of attention distribution causes VLMs to favor textual knowledge in the case of multimodal knowledge conflicts, resulting in differences from the image information. In this paper, we propose Re-Balancing Contrastive Decoding (RBD) method, which employs textual and visual branches to recalibrate attention distribution in VLMs. Specifically, the textual branch injects image noise to stimulate the model's dependency on text, thereby reducing textual bias. Concurrently, the visual branch focuses on the selection of significant tokens, refining the attention mechanism to highlight the primary subject. This dual-branch strategy enables the RBD method to diminish textual bias while enhancing visual information. Experimental results demonstrate that our method, RBD, outperforms the existing methods by the CHAIR and POPE metrics, mitigate hallucinations without reducing the model's general capabilities. |
| title | Mitigating Hallucination in Visual-Language Models via Re-Balancing Contrastive Decoding |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2409.06485 |