CROPE: Evaluating In-Context Adaptation of Vision and Language Models to Culture-Specific Concepts
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909479572865024 |
|---|---|
| author | Nikandrou, Malvina Pantazopoulos, Georgios Vitsakis, Nikolas Konstas, Ioannis Suglia, Alessandro |
| author_facet | Nikandrou, Malvina Pantazopoulos, Georgios Vitsakis, Nikolas Konstas, Ioannis Suglia, Alessandro |
| contents | As Vision and Language models (VLMs) are reaching users across the globe, assessing their cultural understanding has become a critical challenge. In this paper, we introduce CROPE, a visual question answering benchmark designed to probe the knowledge of culture-specific concepts and evaluate the capacity for cultural adaptation through contextual information. This allows us to distinguish between parametric knowledge acquired during training and contextual knowledge provided during inference via visual and textual descriptions. Our evaluation of several state-of-the-art open VLMs shows large performance disparities between culture-specific and common concepts in the parametric setting. Moreover, experiments with contextual knowledge indicate that models struggle to effectively utilize multimodal information and bind culture-specific concepts to their depictions. Our findings reveal limitations in the cultural understanding and adaptability of current VLMs that need to be addressed toward more culturally inclusive models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_15453 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | CROPE: Evaluating In-Context Adaptation of Vision and Language Models to Culture-Specific Concepts Nikandrou, Malvina Pantazopoulos, Georgios Vitsakis, Nikolas Konstas, Ioannis Suglia, Alessandro Computation and Language Computer Vision and Pattern Recognition As Vision and Language models (VLMs) are reaching users across the globe, assessing their cultural understanding has become a critical challenge. In this paper, we introduce CROPE, a visual question answering benchmark designed to probe the knowledge of culture-specific concepts and evaluate the capacity for cultural adaptation through contextual information. This allows us to distinguish between parametric knowledge acquired during training and contextual knowledge provided during inference via visual and textual descriptions. Our evaluation of several state-of-the-art open VLMs shows large performance disparities between culture-specific and common concepts in the parametric setting. Moreover, experiments with contextual knowledge indicate that models struggle to effectively utilize multimodal information and bind culture-specific concepts to their depictions. Our findings reveal limitations in the cultural understanding and adaptability of current VLMs that need to be addressed toward more culturally inclusive models. |
| title | CROPE: Evaluating In-Context Adaptation of Vision and Language Models to Culture-Specific Concepts |
| topic | Computation and Language Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2410.15453 |