Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2508.00673 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916875707875328 |
|---|---|
| author | Farsi, Farhan Aghababaloo, Farnaz Motlagh, Shahriar Shariati Ghofrani, Parsa SadraeiJavaheri, MohammadAli Bali, Shayan Shabani, Amirhossein Bijary, Farbod Zamaninejad, Ghazal Salehoof, AmirMohammad Momtazi, Saeedeh |
| author_facet | Farsi, Farhan Aghababaloo, Farnaz Motlagh, Shahriar Shariati Ghofrani, Parsa SadraeiJavaheri, MohammadAli Bali, Shayan Shabani, Amirhossein Bijary, Farbod Zamaninejad, Ghazal Salehoof, AmirMohammad Momtazi, Saeedeh |
| contents | As large language models (LLMs) become increasingly embedded in our daily lives, evaluating their quality and reliability across diverse contexts has become essential. While comprehensive benchmarks exist for assessing LLM performance in English, there remains a significant gap in evaluation resources for other languages. Moreover, because most LLMs are trained primarily on data rooted in European and American cultures, they often lack familiarity with non-Western cultural contexts. To address this limitation, our study focuses on the Persian language and Iranian culture. We introduce 19 new evaluation datasets specifically designed to assess LLMs on topics such as Iranian law, Persian grammar, Persian idioms, and university entrance exams. Using these datasets, we benchmarked 41 prominent LLMs, aiming to bridge the existing cultural and linguistic evaluation gap in the field. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_00673 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | MELAC: Massive Evaluation of Large Language Models with Alignment of Culture in Persian Language Farsi, Farhan Aghababaloo, Farnaz Motlagh, Shahriar Shariati Ghofrani, Parsa SadraeiJavaheri, MohammadAli Bali, Shayan Shabani, Amirhossein Bijary, Farbod Zamaninejad, Ghazal Salehoof, AmirMohammad Momtazi, Saeedeh Computation and Language As large language models (LLMs) become increasingly embedded in our daily lives, evaluating their quality and reliability across diverse contexts has become essential. While comprehensive benchmarks exist for assessing LLM performance in English, there remains a significant gap in evaluation resources for other languages. Moreover, because most LLMs are trained primarily on data rooted in European and American cultures, they often lack familiarity with non-Western cultural contexts. To address this limitation, our study focuses on the Persian language and Iranian culture. We introduce 19 new evaluation datasets specifically designed to assess LLMs on topics such as Iranian law, Persian grammar, Persian idioms, and university entrance exams. Using these datasets, we benchmarked 41 prominent LLMs, aiming to bridge the existing cultural and linguistic evaluation gap in the field. |
| title | MELAC: Massive Evaluation of Large Language Models with Alignment of Culture in Persian Language |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2508.00673 |