Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2501.01282 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917882910212096 |
|---|---|
| author | Liu, Shudong Jin, Yiqiao Li, Cheng Wong, Derek F. Wen, Qingsong Sun, Lichao Chen, Haipeng Xie, Xing Wang, Jindong |
| author_facet | Liu, Shudong Jin, Yiqiao Li, Cheng Wong, Derek F. Wen, Qingsong Sun, Lichao Chen, Haipeng Xie, Xing Wang, Jindong |
| contents | Vision-language models (VLMs) have advanced human-AI interaction but struggle with cultural understanding, often misinterpreting symbols, gestures, and artifacts due to biases in predominantly Western-centric training data. In this paper, we construct CultureVerse, a large-scale multimodal benchmark covering 19, 682 cultural concepts, 188 countries/regions, 15 cultural concepts, and 3 question types, with the aim of characterizing and improving VLMs' multicultural understanding capabilities. Then, we propose CultureVLM, a series of VLMs fine-tuned on our dataset to achieve significant performance improvement in cultural understanding. Our evaluation of 16 models reveals significant disparities, with a stronger performance in Western concepts and weaker results in African and Asian contexts. Fine-tuning on our CultureVerse enhances cultural perception, demonstrating cross-cultural, cross-continent, and cross-dataset generalization without sacrificing performance on models' general VLM benchmarks. We further present insights on cultural generalization and forgetting. We hope that this work could lay the foundation for more equitable and culturally aware multimodal AI systems. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_01282 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | CultureVLM: Characterizing and Improving Cultural Understanding of Vision-Language Models for over 100 Countries Liu, Shudong Jin, Yiqiao Li, Cheng Wong, Derek F. Wen, Qingsong Sun, Lichao Chen, Haipeng Xie, Xing Wang, Jindong Artificial Intelligence Computation and Language Computer Vision and Pattern Recognition Vision-language models (VLMs) have advanced human-AI interaction but struggle with cultural understanding, often misinterpreting symbols, gestures, and artifacts due to biases in predominantly Western-centric training data. In this paper, we construct CultureVerse, a large-scale multimodal benchmark covering 19, 682 cultural concepts, 188 countries/regions, 15 cultural concepts, and 3 question types, with the aim of characterizing and improving VLMs' multicultural understanding capabilities. Then, we propose CultureVLM, a series of VLMs fine-tuned on our dataset to achieve significant performance improvement in cultural understanding. Our evaluation of 16 models reveals significant disparities, with a stronger performance in Western concepts and weaker results in African and Asian contexts. Fine-tuning on our CultureVerse enhances cultural perception, demonstrating cross-cultural, cross-continent, and cross-dataset generalization without sacrificing performance on models' general VLM benchmarks. We further present insights on cultural generalization and forgetting. We hope that this work could lay the foundation for more equitable and culturally aware multimodal AI systems. |
| title | CultureVLM: Characterizing and Improving Cultural Understanding of Vision-Language Models for over 100 Countries |
| topic | Artificial Intelligence Computation and Language Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2501.01282 |