Enhancing Visual-Language Modality Alignment in Large Vision Language Models via Self-Improvement
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866929704527724544 |
|---|---|
| author | Wang, Xiyao Chen, Jiuhai Wang, Zhaoyang Zhou, Yuhang Zhou, Yiyang Yao, Huaxiu Zhou, Tianyi Goldstein, Tom Bhatia, Parminder Huang, Furong Xiao, Cao |
| author_facet | Wang, Xiyao Chen, Jiuhai Wang, Zhaoyang Zhou, Yuhang Zhou, Yiyang Yao, Huaxiu Zhou, Tianyi Goldstein, Tom Bhatia, Parminder Huang, Furong Xiao, Cao |
| contents | Large vision-language models (LVLMs) have achieved impressive results in visual question-answering and reasoning tasks through vision instruction tuning on specific datasets. However, there remains significant room for improvement in aligning visual and language modalities. Existing methods often depend on external models or data, leading to uncontrollable and unstable alignment results. In this paper, we propose SIMA, a self-improvement framework that enhances visual and language modality alignment without external dependencies. SIMA leverages existing vision instruction tuning datasets to self-generate responses, incorporating an in-context self-critic mechanism that constructs preference pairs for tuning. Crucially, our approach allows LVLMs to act as critics by designing effective critic prompts, eliminating the need for additional fine-tuning with external instruction data. We introduce three novel visual metrics within the self-critic process to guide judgment, significantly improving the accuracy of self-critic. Through extensive experiments across 14 hallucination and comprehensive benchmarks, we demonstrate that SIMA significantly improves LVLM's performance and outperforms previous approaches, achieving superior modality alignment. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_15973 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Enhancing Visual-Language Modality Alignment in Large Vision Language Models via Self-Improvement Wang, Xiyao Chen, Jiuhai Wang, Zhaoyang Zhou, Yuhang Zhou, Yiyang Yao, Huaxiu Zhou, Tianyi Goldstein, Tom Bhatia, Parminder Huang, Furong Xiao, Cao Computer Vision and Pattern Recognition Artificial Intelligence Computation and Language Machine Learning Large vision-language models (LVLMs) have achieved impressive results in visual question-answering and reasoning tasks through vision instruction tuning on specific datasets. However, there remains significant room for improvement in aligning visual and language modalities. Existing methods often depend on external models or data, leading to uncontrollable and unstable alignment results. In this paper, we propose SIMA, a self-improvement framework that enhances visual and language modality alignment without external dependencies. SIMA leverages existing vision instruction tuning datasets to self-generate responses, incorporating an in-context self-critic mechanism that constructs preference pairs for tuning. Crucially, our approach allows LVLMs to act as critics by designing effective critic prompts, eliminating the need for additional fine-tuning with external instruction data. We introduce three novel visual metrics within the self-critic process to guide judgment, significantly improving the accuracy of self-critic. Through extensive experiments across 14 hallucination and comprehensive benchmarks, we demonstrate that SIMA significantly improves LVLM's performance and outperforms previous approaches, achieving superior modality alignment. |
| title | Enhancing Visual-Language Modality Alignment in Large Vision Language Models via Self-Improvement |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2405.15973 |