PerCul: A Story-Driven Cultural Evaluation of LLMs in Persian
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866912490101669888 |
|---|---|
| author | Monazzah, Erfan Moosavi Rahimzadeh, Vahid Yaghoobzadeh, Yadollah Shakery, Azadeh Pilehvar, Mohammad Taher |
| author_facet | Monazzah, Erfan Moosavi Rahimzadeh, Vahid Yaghoobzadeh, Yadollah Shakery, Azadeh Pilehvar, Mohammad Taher |
| contents | Large language models predominantly reflect Western cultures, largely due to the dominance of English-centric training data. This imbalance presents a significant challenge, as LLMs are increasingly used across diverse contexts without adequate evaluation of their cultural competence in non-English languages, including Persian. To address this gap, we introduce PerCul, a carefully constructed dataset designed to assess the sensitivity of LLMs toward Persian culture. PerCul features story-based, multiple-choice questions that capture culturally nuanced scenarios. Unlike existing benchmarks, PerCul is curated with input from native Persian annotators to ensure authenticity and to prevent the use of translation as a shortcut. We evaluate several state-of-the-art multilingual and Persian-specific LLMs, establishing a foundation for future research in cross-cultural NLP evaluation. Our experiments demonstrate a 11.3% gap between best closed source model and layperson baseline while the gap increases to 21.3% by using the best open-weight model. You can access the dataset from here: https://huggingface.co/datasets/teias-ai/percul |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_07459 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | PerCul: A Story-Driven Cultural Evaluation of LLMs in Persian Monazzah, Erfan Moosavi Rahimzadeh, Vahid Yaghoobzadeh, Yadollah Shakery, Azadeh Pilehvar, Mohammad Taher Computation and Language Artificial Intelligence Computers and Society I.2.7 Large language models predominantly reflect Western cultures, largely due to the dominance of English-centric training data. This imbalance presents a significant challenge, as LLMs are increasingly used across diverse contexts without adequate evaluation of their cultural competence in non-English languages, including Persian. To address this gap, we introduce PerCul, a carefully constructed dataset designed to assess the sensitivity of LLMs toward Persian culture. PerCul features story-based, multiple-choice questions that capture culturally nuanced scenarios. Unlike existing benchmarks, PerCul is curated with input from native Persian annotators to ensure authenticity and to prevent the use of translation as a shortcut. We evaluate several state-of-the-art multilingual and Persian-specific LLMs, establishing a foundation for future research in cross-cultural NLP evaluation. Our experiments demonstrate a 11.3% gap between best closed source model and layperson baseline while the gap increases to 21.3% by using the best open-weight model. You can access the dataset from here: https://huggingface.co/datasets/teias-ai/percul |
| title | PerCul: A Story-Driven Cultural Evaluation of LLMs in Persian |
| topic | Computation and Language Artificial Intelligence Computers and Society I.2.7 |
| url | https://arxiv.org/abs/2502.07459 |