Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2407.15487 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916331659460608 |
|---|---|
| author | Nulli, Matteo Ibrahimi, Anesa Pal, Avik Lee, Hoshe Najdenkoska, Ivona |
| author_facet | Nulli, Matteo Ibrahimi, Anesa Pal, Avik Lee, Hoshe Najdenkoska, Ivona |
| contents | Vision-Language Models (VLMs) have shown remarkable capabilities in a large number of downstream tasks. Nonetheless, compositional image understanding remains a rather difficult task due to the object bias present in training data. In this work, we investigate the reasons for such a lack of capability by performing an extensive bench-marking of compositional understanding in VLMs. We compare contrastive models with generative ones and analyze their differences in architecture, pre-training data, and training tasks and losses. Furthermore, we leverage In-Context Learning (ICL) as a way to improve the ability of VLMs to perform more complex reasoning and understanding given an image. Our extensive experiments demonstrate that our proposed approach outperforms baseline models across multiple compositional understanding datasets. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_15487 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | In-Context Learning Improves Compositional Understanding of Vision-Language Models Nulli, Matteo Ibrahimi, Anesa Pal, Avik Lee, Hoshe Najdenkoska, Ivona Computer Vision and Pattern Recognition Artificial Intelligence Vision-Language Models (VLMs) have shown remarkable capabilities in a large number of downstream tasks. Nonetheless, compositional image understanding remains a rather difficult task due to the object bias present in training data. In this work, we investigate the reasons for such a lack of capability by performing an extensive bench-marking of compositional understanding in VLMs. We compare contrastive models with generative ones and analyze their differences in architecture, pre-training data, and training tasks and losses. Furthermore, we leverage In-Context Learning (ICL) as a way to improve the ability of VLMs to perform more complex reasoning and understanding given an image. Our extensive experiments demonstrate that our proposed approach outperforms baseline models across multiple compositional understanding datasets. |
| title | In-Context Learning Improves Compositional Understanding of Vision-Language Models |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence |
| url | https://arxiv.org/abs/2407.15487 |