Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2602.22827 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910053521424384 |
|---|---|
| author | Iranmanesh, Reihaneh Davoudi, Saeedeh Abrishamchian, Pasha Frieder, Ophir Goharian, Nazli |
| author_facet | Iranmanesh, Reihaneh Davoudi, Saeedeh Abrishamchian, Pasha Frieder, Ophir Goharian, Nazli |
| contents | This paper presents a comprehensive evaluation framework for assessing the cultural competence of large language models (LLMs) in Persian. Existing Persian cultural benchmarks rely predominantly on multiple-choice formats and English-centric metrics that fail to capture Persian's morphological complexity and semantic nuance. Our framework introduces a Persian-specific short-answer evaluation that combines rule-based morphological normalization with a hybrid syntactic and semantic similarity module, enabling robust soft-match scoring beyond exact string overlap. Through systematic evaluation of 15 state-of-the-art open- and closed-source models across three culturally grounded Persian datasets, we demonstrate that our hybrid evaluation improves scoring consistency by +10 compared to exact-match baselines by capturing meaning that surface-level methods cannot detect. Our human evaluation further confirms that the proposed semantic similarity metric achieves higher agreement with human judgments than LLM-based judges. We publicly release our evaluation framework, providing the first standardized benchmark for measuring cultural understanding in Persian and establishing a reproducible foundation for cross-cultural LLM evaluation research. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_22827 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | TARAZ: Persian Short-Answer Question Benchmark for Cultural Evaluation of Language Models Iranmanesh, Reihaneh Davoudi, Saeedeh Abrishamchian, Pasha Frieder, Ophir Goharian, Nazli Computation and Language Machine Learning This paper presents a comprehensive evaluation framework for assessing the cultural competence of large language models (LLMs) in Persian. Existing Persian cultural benchmarks rely predominantly on multiple-choice formats and English-centric metrics that fail to capture Persian's morphological complexity and semantic nuance. Our framework introduces a Persian-specific short-answer evaluation that combines rule-based morphological normalization with a hybrid syntactic and semantic similarity module, enabling robust soft-match scoring beyond exact string overlap. Through systematic evaluation of 15 state-of-the-art open- and closed-source models across three culturally grounded Persian datasets, we demonstrate that our hybrid evaluation improves scoring consistency by +10 compared to exact-match baselines by capturing meaning that surface-level methods cannot detect. Our human evaluation further confirms that the proposed semantic similarity metric achieves higher agreement with human judgments than LLM-based judges. We publicly release our evaluation framework, providing the first standardized benchmark for measuring cultural understanding in Persian and establishing a reproducible foundation for cross-cultural LLM evaluation research. |
| title | TARAZ: Persian Short-Answer Question Benchmark for Cultural Evaluation of Language Models |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2602.22827 |