XCR-Bench: A Multi-Task Benchmark for Evaluating Cultural Reasoning in LLMs
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918296934154240 |
|---|---|
| author | Kabir, Mohsinul Ahmed, Tasnim Rahman, Md Mezbaur Ji, Shaoxiong Alhuzali, Hassan Ananiadou, Sophia |
| author_facet | Kabir, Mohsinul Ahmed, Tasnim Rahman, Md Mezbaur Ji, Shaoxiong Alhuzali, Hassan Ananiadou, Sophia |
| contents | Cross-cultural competence in large language models (LLMs) requires the ability to identify Culture-Specific Items (CSIs) and to adapt them appropriately across cultural contexts. Progress in evaluating this capability has been constrained by the scarcity of high-quality CSI-annotated corpora with parallel cross-cultural sentence pairs. To address this limitation, we introduce XCR-Bench, a Cross(X)-Cultural Reasoning Benchmark consisting of 4.9k parallel sentences and 1,098 unique CSIs, spanning three distinct reasoning tasks with corresponding evaluation metrics. Our corpus integrates Newmark's CSI framework with Hall's Triad of Culture, enabling systematic analysis of cultural reasoning beyond surface-level artifacts and into semi-visible and invisible cultural elements such as social norms, beliefs, and values. Our findings show that state-of-the-art LLMs exhibit consistent weaknesses in identifying and adapting CSIs related to social etiquette and cultural reference. Additionally, we find evidence that LLMs encode regional and ethno-religious biases even within a single linguistic setting during cultural adaptation. We release our corpus and code to facilitate future research on cross-cultural NLP. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_14063 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | XCR-Bench: A Multi-Task Benchmark for Evaluating Cultural Reasoning in LLMs Kabir, Mohsinul Ahmed, Tasnim Rahman, Md Mezbaur Ji, Shaoxiong Alhuzali, Hassan Ananiadou, Sophia Computation and Language Artificial Intelligence Computers and Society Cross-cultural competence in large language models (LLMs) requires the ability to identify Culture-Specific Items (CSIs) and to adapt them appropriately across cultural contexts. Progress in evaluating this capability has been constrained by the scarcity of high-quality CSI-annotated corpora with parallel cross-cultural sentence pairs. To address this limitation, we introduce XCR-Bench, a Cross(X)-Cultural Reasoning Benchmark consisting of 4.9k parallel sentences and 1,098 unique CSIs, spanning three distinct reasoning tasks with corresponding evaluation metrics. Our corpus integrates Newmark's CSI framework with Hall's Triad of Culture, enabling systematic analysis of cultural reasoning beyond surface-level artifacts and into semi-visible and invisible cultural elements such as social norms, beliefs, and values. Our findings show that state-of-the-art LLMs exhibit consistent weaknesses in identifying and adapting CSIs related to social etiquette and cultural reference. Additionally, we find evidence that LLMs encode regional and ethno-religious biases even within a single linguistic setting during cultural adaptation. We release our corpus and code to facilitate future research on cross-cultural NLP. |
| title | XCR-Bench: A Multi-Task Benchmark for Evaluating Cultural Reasoning in LLMs |
| topic | Computation and Language Artificial Intelligence Computers and Society |
| url | https://arxiv.org/abs/2601.14063 |