Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2410.00905 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910627622027264 |
|---|---|
| author | Li, Yuheng Liu, Haotian Cai, Mu Li, Yijun Shechtman, Eli Lin, Zhe Lee, Yong Jae Singh, Krishna Kumar |
| author_facet | Li, Yuheng Liu, Haotian Cai, Mu Li, Yijun Shechtman, Eli Lin, Zhe Lee, Yong Jae Singh, Krishna Kumar |
| contents | In this paper, we introduce a model designed to improve the prediction of image-text alignment, targeting the challenge of compositional understanding in current visual-language models. Our approach focuses on generating high-quality training datasets for the alignment task by producing mixed-type negative captions derived from positive ones. Critically, we address the distribution imbalance between positive and negative captions to ensure that the alignment model does not depend solely on textual information but also considers the associated images for predicting alignment accurately. By creating this enhanced training data, we fine-tune an existing leading visual-language model to boost its capability in understanding alignment. Our model significantly outperforms current top-performing methods across various datasets. We also demonstrate the applicability of our model by ranking the images generated by text-to-image models based on text alignment. Project page: \url{https://yuheng-li.github.io/LLaVA-score/} |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_00905 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Removing Distributional Discrepancies in Captions Improves Image-Text Alignment Li, Yuheng Liu, Haotian Cai, Mu Li, Yijun Shechtman, Eli Lin, Zhe Lee, Yong Jae Singh, Krishna Kumar Computer Vision and Pattern Recognition In this paper, we introduce a model designed to improve the prediction of image-text alignment, targeting the challenge of compositional understanding in current visual-language models. Our approach focuses on generating high-quality training datasets for the alignment task by producing mixed-type negative captions derived from positive ones. Critically, we address the distribution imbalance between positive and negative captions to ensure that the alignment model does not depend solely on textual information but also considers the associated images for predicting alignment accurately. By creating this enhanced training data, we fine-tune an existing leading visual-language model to boost its capability in understanding alignment. Our model significantly outperforms current top-performing methods across various datasets. We also demonstrate the applicability of our model by ranking the images generated by text-to-image models based on text alignment. Project page: \url{https://yuheng-li.github.io/LLaVA-score/} |
| title | Removing Distributional Discrepancies in Captions Improves Image-Text Alignment |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2410.00905 |