Towards Mitigating Hallucinations in Large Vision-Language Models by Refining Textual Embeddings
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866909892146626560 |
|---|---|
| author | Agrawal, Aakriti KV, Gouthaman Aralikatti, Rohith Jagatap, Gauri Yuan, Jiaxin Kamarshi, Vijay Fanelli, Andrea Huang, Furong |
| author_facet | Agrawal, Aakriti KV, Gouthaman Aralikatti, Rohith Jagatap, Gauri Yuan, Jiaxin Kamarshi, Vijay Fanelli, Andrea Huang, Furong |
| contents | In this work, we identify an inherent bias in prevailing LVLM architectures toward the language modality, largely resulting from the common practice of simply appending visual embeddings to the input text sequence. To address this, we propose a simple yet effective method that refines textual embeddings by integrating average-pooled visual features. Our approach demonstrably improves visual grounding and significantly reduces hallucinations on established benchmarks. While average pooling offers a straightforward, robust, and efficient means of incorporating visual information, we believe that more sophisticated fusion methods could further enhance visual grounding and cross-modal alignment. Given that the primary focus of this work is to highlight the modality imbalance and its impact on hallucinations -- and to show that refining textual embeddings with visual information mitigates this issue -- we leave exploration of advanced fusion strategies for future work. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_05017 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Towards Mitigating Hallucinations in Large Vision-Language Models by Refining Textual Embeddings Agrawal, Aakriti KV, Gouthaman Aralikatti, Rohith Jagatap, Gauri Yuan, Jiaxin Kamarshi, Vijay Fanelli, Andrea Huang, Furong Computer Vision and Pattern Recognition Computation and Language In this work, we identify an inherent bias in prevailing LVLM architectures toward the language modality, largely resulting from the common practice of simply appending visual embeddings to the input text sequence. To address this, we propose a simple yet effective method that refines textual embeddings by integrating average-pooled visual features. Our approach demonstrably improves visual grounding and significantly reduces hallucinations on established benchmarks. While average pooling offers a straightforward, robust, and efficient means of incorporating visual information, we believe that more sophisticated fusion methods could further enhance visual grounding and cross-modal alignment. Given that the primary focus of this work is to highlight the modality imbalance and its impact on hallucinations -- and to show that refining textual embeddings with visual information mitigates this issue -- we leave exploration of advanced fusion strategies for future work. |
| title | Towards Mitigating Hallucinations in Large Vision-Language Models by Refining Textual Embeddings |
| topic | Computer Vision and Pattern Recognition Computation and Language |
| url | https://arxiv.org/abs/2511.05017 |