Enregistré dans:
| Auteurs principaux: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | https://arxiv.org/abs/2509.16188 |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866915503344189440 |
|---|---|
| author | Zhang, Jinghao Jiang, Sihang Guo, Shiwei Chen, Shisong Xiao, Yanghua Feng, Hongwei Liang, Jiaqing HE, Minggui Tao, Shimin Ma, Hongxia |
| author_facet | Zhang, Jinghao Jiang, Sihang Guo, Shiwei Chen, Shisong Xiao, Yanghua Feng, Hongwei Liang, Jiaqing HE, Minggui Tao, Shimin Ma, Hongxia |
| contents | As large language models (LLMs) are increasingly deployed in diverse cultural environments, evaluating their cultural understanding capability has become essential for ensuring trustworthy and culturally aligned applications. However, most existing benchmarks lack comprehensiveness and are challenging to scale and adapt across different cultural contexts, because their frameworks often lack guidance from well-established cultural theories and tend to rely on expert-driven manual annotations. To address these issues, we propose CultureScope, the most comprehensive evaluation framework to date for assessing cultural understanding in LLMs. Inspired by the cultural iceberg theory, we design a novel dimensional schema for cultural knowledge classification, comprising 3 layers and 140 dimensions, which guides the automated construction of culture-specific knowledge bases and corresponding evaluation datasets for any given languages and cultures. Experimental results demonstrate that our method can effectively evaluate cultural understanding. They also reveal that existing large language models lack comprehensive cultural competence, and merely incorporating multilingual data does not necessarily enhance cultural understanding. All code and data files are available at https://github.com/HoganZinger/Culture |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_16188 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | CultureScope: A Dimensional Lens for Probing Cultural Understanding in LLMs Zhang, Jinghao Jiang, Sihang Guo, Shiwei Chen, Shisong Xiao, Yanghua Feng, Hongwei Liang, Jiaqing HE, Minggui Tao, Shimin Ma, Hongxia Computation and Language Artificial Intelligence As large language models (LLMs) are increasingly deployed in diverse cultural environments, evaluating their cultural understanding capability has become essential for ensuring trustworthy and culturally aligned applications. However, most existing benchmarks lack comprehensiveness and are challenging to scale and adapt across different cultural contexts, because their frameworks often lack guidance from well-established cultural theories and tend to rely on expert-driven manual annotations. To address these issues, we propose CultureScope, the most comprehensive evaluation framework to date for assessing cultural understanding in LLMs. Inspired by the cultural iceberg theory, we design a novel dimensional schema for cultural knowledge classification, comprising 3 layers and 140 dimensions, which guides the automated construction of culture-specific knowledge bases and corresponding evaluation datasets for any given languages and cultures. Experimental results demonstrate that our method can effectively evaluate cultural understanding. They also reveal that existing large language models lack comprehensive cultural competence, and merely incorporating multilingual data does not necessarily enhance cultural understanding. All code and data files are available at https://github.com/HoganZinger/Culture |
| title | CultureScope: A Dimensional Lens for Probing Cultural Understanding in LLMs |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2509.16188 |