TCC-Bench: Benchmarking the Traditional Chinese Culture Understanding Capabilities of MLLMs
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866917116765011968 |
|---|---|
| author | Xu, Pengju Wang, Yan Zhang, Shuyuan Zhou, Xuan Li, Xin Yuan, Yue Li, Fengzhao Zhou, Shunyuan Wang, Xingyu Zhang, Yi Zhao, Haiying |
| author_facet | Xu, Pengju Wang, Yan Zhang, Shuyuan Zhou, Xuan Li, Xin Yuan, Yue Li, Fengzhao Zhou, Shunyuan Wang, Xingyu Zhang, Yi Zhao, Haiying |
| contents | Recent progress in Multimodal Large Language Models (MLLMs) have significantly enhanced the ability of artificial intelligence systems to understand and generate multimodal content. However, these models often exhibit limited effectiveness when applied to non-Western cultural contexts, which raises concerns about their wider applicability. To address this limitation, we propose the Traditional Chinese Culture understanding Benchmark (TCC-Bench), a bilingual (i.e., Chinese and English) Visual Question Answering (VQA) benchmark specifically designed for assessing the understanding of traditional Chinese culture by MLLMs. TCC-Bench comprises culturally rich and visually diverse data, incorporating images from museum artifacts, everyday life scenes, comics, and other culturally significant contexts. We adopt a semi-automated pipeline that utilizes GPT-4o in text-only mode to generate candidate questions, followed by human curation to ensure data quality and avoid potential data leakage. The benchmark also avoids language bias by preventing direct disclosure of cultural concepts within question texts. Experimental evaluations across a wide range of MLLMs demonstrate that current models still face significant challenges when reasoning about culturally grounded visual content. The results highlight the need for further research in developing culturally inclusive and context-aware multimodal systems. The code and data can be found at: https://tcc-bench.github.io/. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_11275 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | TCC-Bench: Benchmarking the Traditional Chinese Culture Understanding Capabilities of MLLMs Xu, Pengju Wang, Yan Zhang, Shuyuan Zhou, Xuan Li, Xin Yuan, Yue Li, Fengzhao Zhou, Shunyuan Wang, Xingyu Zhang, Yi Zhao, Haiying Multimedia Artificial Intelligence Computers and Society Recent progress in Multimodal Large Language Models (MLLMs) have significantly enhanced the ability of artificial intelligence systems to understand and generate multimodal content. However, these models often exhibit limited effectiveness when applied to non-Western cultural contexts, which raises concerns about their wider applicability. To address this limitation, we propose the Traditional Chinese Culture understanding Benchmark (TCC-Bench), a bilingual (i.e., Chinese and English) Visual Question Answering (VQA) benchmark specifically designed for assessing the understanding of traditional Chinese culture by MLLMs. TCC-Bench comprises culturally rich and visually diverse data, incorporating images from museum artifacts, everyday life scenes, comics, and other culturally significant contexts. We adopt a semi-automated pipeline that utilizes GPT-4o in text-only mode to generate candidate questions, followed by human curation to ensure data quality and avoid potential data leakage. The benchmark also avoids language bias by preventing direct disclosure of cultural concepts within question texts. Experimental evaluations across a wide range of MLLMs demonstrate that current models still face significant challenges when reasoning about culturally grounded visual content. The results highlight the need for further research in developing culturally inclusive and context-aware multimodal systems. The code and data can be found at: https://tcc-bench.github.io/. |
| title | TCC-Bench: Benchmarking the Traditional Chinese Culture Understanding Capabilities of MLLMs |
| topic | Multimedia Artificial Intelligence Computers and Society |
| url | https://arxiv.org/abs/2505.11275 |