Sensitivity of Generative VLMs to Semantically and Lexically Altered Prompts
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866929547448942592 |
|---|---|
| author | Dumpala, Sri Harsha Jaiswal, Aman Sastry, Chandramouli Milios, Evangelos Oore, Sageev Sajjad, Hassan |
| author_facet | Dumpala, Sri Harsha Jaiswal, Aman Sastry, Chandramouli Milios, Evangelos Oore, Sageev Sajjad, Hassan |
| contents | Despite the significant influx of prompt-tuning techniques for generative vision-language models (VLMs), it remains unclear how sensitive these models are to lexical and semantic alterations in prompts. In this paper, we evaluate the ability of generative VLMs to understand lexical and semantic changes in text using the SugarCrepe++ dataset. We analyze the sensitivity of VLMs to lexical alterations in prompts without corresponding semantic changes. Our findings demonstrate that generative VLMs are highly sensitive to such alterations. Additionally, we show that this vulnerability affects the performance of techniques aimed at achieving consistency in their outputs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_13030 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Sensitivity of Generative VLMs to Semantically and Lexically Altered Prompts Dumpala, Sri Harsha Jaiswal, Aman Sastry, Chandramouli Milios, Evangelos Oore, Sageev Sajjad, Hassan Computer Vision and Pattern Recognition Computation and Language Machine Learning Despite the significant influx of prompt-tuning techniques for generative vision-language models (VLMs), it remains unclear how sensitive these models are to lexical and semantic alterations in prompts. In this paper, we evaluate the ability of generative VLMs to understand lexical and semantic changes in text using the SugarCrepe++ dataset. We analyze the sensitivity of VLMs to lexical alterations in prompts without corresponding semantic changes. Our findings demonstrate that generative VLMs are highly sensitive to such alterations. Additionally, we show that this vulnerability affects the performance of techniques aimed at achieving consistency in their outputs. |
| title | Sensitivity of Generative VLMs to Semantically and Lexically Altered Prompts |
| topic | Computer Vision and Pattern Recognition Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2410.13030 |