On the Limitations of Vision-Language Models in Understanding Image Transforms
Fuente:
arXiv
Salvato in:
| Autori principali: | , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866915197044654080 |
|---|---|
| author | Anis, Ahmad Mustafa Ali, Hasnain Sarfraz, Saquib |
| author_facet | Anis, Ahmad Mustafa Ali, Hasnain Sarfraz, Saquib |
| contents | Vision Language Models (VLMs) have demonstrated significant potential in various downstream tasks, including Image/Video Generation, Visual Question Answering, Multimodal Chatbots, and Video Understanding. However, these models often struggle with basic image transformations. This paper investigates the image-level understanding of VLMs, specifically CLIP by OpenAI and SigLIP by Google. Our findings reveal that these models lack comprehension of multiple image-level augmentations. To facilitate this study, we created an augmented version of the Flickr8k dataset, pairing each image with a detailed description of the applied transformation. We further explore how this deficiency impacts downstream tasks, particularly in image editing, and evaluate the performance of state-of-the-art Image2Image models on simple transformations. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_09837 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | On the Limitations of Vision-Language Models in Understanding Image Transforms Anis, Ahmad Mustafa Ali, Hasnain Sarfraz, Saquib Computer Vision and Pattern Recognition Artificial Intelligence Computation and Language I.4; I.2.10; I.2.7 Vision Language Models (VLMs) have demonstrated significant potential in various downstream tasks, including Image/Video Generation, Visual Question Answering, Multimodal Chatbots, and Video Understanding. However, these models often struggle with basic image transformations. This paper investigates the image-level understanding of VLMs, specifically CLIP by OpenAI and SigLIP by Google. Our findings reveal that these models lack comprehension of multiple image-level augmentations. To facilitate this study, we created an augmented version of the Flickr8k dataset, pairing each image with a detailed description of the applied transformation. We further explore how this deficiency impacts downstream tasks, particularly in image editing, and evaluate the performance of state-of-the-art Image2Image models on simple transformations. |
| title | On the Limitations of Vision-Language Models in Understanding Image Transforms |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Computation and Language I.4; I.2.10; I.2.7 |
| url | https://arxiv.org/abs/2503.09837 |