Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2510.05291 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918525288841216 |
|---|---|
| author | Naous, Tarek Savit, Anagha Catalan, Carlos Rafael Guo, Geyang Lee, Jaehyeok Lee, Kyungdon Dizon, Lheane Marie Ye, Mengyu Kothari, Neel Singh, Sahajpreet Masud, Sarah Patwa, Tanish Tran, Trung Thanh Khan, Zohaib Ritter, Alan Chakraborty, Tanmoy Arase, Yuki Sakaguchi, Keisuke Bak, JinYeong Xu, Wei |
| author_facet | Naous, Tarek Savit, Anagha Catalan, Carlos Rafael Guo, Geyang Lee, Jaehyeok Lee, Kyungdon Dizon, Lheane Marie Ye, Mengyu Kothari, Neel Singh, Sahajpreet Masud, Sarah Patwa, Tanish Tran, Trung Thanh Khan, Zohaib Ritter, Alan Chakraborty, Tanmoy Arase, Yuki Sakaguchi, Keisuke Bak, JinYeong Xu, Wei |
| contents | As Large Language Models (LLMs) develop stronger multilingual capabilities, their sensitivity to culturally diverse entities becomes increasingly important. Prior work by Naous et al. (2024) has shown that LLMs often favor Western-associated entities in Arabic. Due to the lack of entity-centric multilingual benchmarks, it remains unclear if such biases also manifest in various non-Western languages. In this paper, we introduce Camellia, a benchmark for evaluating entity-centric cultural biases in nine Asian languages, spanning six Asian cultures. Camellia includes 19,530 manually annotated entities associated with the covered Asian or Western cultures, as well as 2,173 masked contexts for these entities derived from social media posts. Using Camellia, we evaluate cultural biases in four recent multilingual LLMs across three tasks: cultural context adaptation, sentiment association, and entity extractive QA. Our analyses show that LLMs struggle with cultural adaptation across these languages, with performance differing across models developed in different regions. We further observe that different LLM families can hold distinct biases, reflected in the ways they link cultures to particular sentiments. Lastly, we find that LLMs can struggle with context understanding in some Asian languages, creating performance gaps between cultures in entity extraction. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_05291 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Camellia: Benchmarking Cultural Biases in LLMs for Asian Languages Naous, Tarek Savit, Anagha Catalan, Carlos Rafael Guo, Geyang Lee, Jaehyeok Lee, Kyungdon Dizon, Lheane Marie Ye, Mengyu Kothari, Neel Singh, Sahajpreet Masud, Sarah Patwa, Tanish Tran, Trung Thanh Khan, Zohaib Ritter, Alan Chakraborty, Tanmoy Arase, Yuki Sakaguchi, Keisuke Bak, JinYeong Xu, Wei Computation and Language As Large Language Models (LLMs) develop stronger multilingual capabilities, their sensitivity to culturally diverse entities becomes increasingly important. Prior work by Naous et al. (2024) has shown that LLMs often favor Western-associated entities in Arabic. Due to the lack of entity-centric multilingual benchmarks, it remains unclear if such biases also manifest in various non-Western languages. In this paper, we introduce Camellia, a benchmark for evaluating entity-centric cultural biases in nine Asian languages, spanning six Asian cultures. Camellia includes 19,530 manually annotated entities associated with the covered Asian or Western cultures, as well as 2,173 masked contexts for these entities derived from social media posts. Using Camellia, we evaluate cultural biases in four recent multilingual LLMs across three tasks: cultural context adaptation, sentiment association, and entity extractive QA. Our analyses show that LLMs struggle with cultural adaptation across these languages, with performance differing across models developed in different regions. We further observe that different LLM families can hold distinct biases, reflected in the ways they link cultures to particular sentiments. Lastly, we find that LLMs can struggle with context understanding in some Asian languages, creating performance gaps between cultures in entity extraction. |
| title | Camellia: Benchmarking Cultural Biases in LLMs for Asian Languages |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2510.05291 |