FarsEval-PKBETS: A new diverse benchmark for evaluating Persian large language models
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913801114222592 |
|---|---|
| author | Shamsfard, Mehrnoush Saaberi, Zahra manesh, Mostafa Karimi Hashemi, Seyed Mohammad Hossein Vatankhah, Zahra Ramezani, Motahareh Pourazin, Niki Zare, Tara Azimi, Maryam Chitsaz, Sarina Khoraminejad, Sama Mortazavi, Morteza Mahdavi Chizari, Mohammad Mahdi Maleki, Sahar Majd, Seyed Soroush Masumi, Mostafa Khoeini, Sayed Ali Musavi Mohseni, Amir Alipour, Sogol |
| author_facet | Shamsfard, Mehrnoush Saaberi, Zahra manesh, Mostafa Karimi Hashemi, Seyed Mohammad Hossein Vatankhah, Zahra Ramezani, Motahareh Pourazin, Niki Zare, Tara Azimi, Maryam Chitsaz, Sarina Khoraminejad, Sama Mortazavi, Morteza Mahdavi Chizari, Mohammad Mahdi Maleki, Sahar Majd, Seyed Soroush Masumi, Mostafa Khoeini, Sayed Ali Musavi Mohseni, Amir Alipour, Sogol |
| contents | Research on evaluating and analyzing large language models (LLMs) has been extensive for resource-rich languages such as English, yet their performance in languages such as Persian has received considerably less attention. This paper introduces FarsEval-PKBETS benchmark, a subset of FarsEval project for evaluating large language models in Persian. This benchmark consists of 4000 questions and answers in various formats, including multiple choice, short answer and descriptive responses. It covers a wide range of domains and tasks,including medicine, law, religion, Persian language, encyclopedic knowledge, human preferences, social knowledge, ethics and bias, text generation, and respecting others' rights. This bechmark incorporates linguistics, cultural, and local considerations relevant to the Persian language and Iran. To ensure the questions are challenging for current LLMs, three models -- Llama3-70B, PersianMind, and Dorna -- were evaluated using this benchmark. Their average accuracy was below 50%, meaning they provided fully correct answers to fewer than half of the questions. These results indicate that current language models are still far from being able to solve this benchmark |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_14690 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | FarsEval-PKBETS: A new diverse benchmark for evaluating Persian large language models Shamsfard, Mehrnoush Saaberi, Zahra manesh, Mostafa Karimi Hashemi, Seyed Mohammad Hossein Vatankhah, Zahra Ramezani, Motahareh Pourazin, Niki Zare, Tara Azimi, Maryam Chitsaz, Sarina Khoraminejad, Sama Mortazavi, Morteza Mahdavi Chizari, Mohammad Mahdi Maleki, Sahar Majd, Seyed Soroush Masumi, Mostafa Khoeini, Sayed Ali Musavi Mohseni, Amir Alipour, Sogol Computation and Language Artificial Intelligence 68T50 I.2.7; E.0 Research on evaluating and analyzing large language models (LLMs) has been extensive for resource-rich languages such as English, yet their performance in languages such as Persian has received considerably less attention. This paper introduces FarsEval-PKBETS benchmark, a subset of FarsEval project for evaluating large language models in Persian. This benchmark consists of 4000 questions and answers in various formats, including multiple choice, short answer and descriptive responses. It covers a wide range of domains and tasks,including medicine, law, religion, Persian language, encyclopedic knowledge, human preferences, social knowledge, ethics and bias, text generation, and respecting others' rights. This bechmark incorporates linguistics, cultural, and local considerations relevant to the Persian language and Iran. To ensure the questions are challenging for current LLMs, three models -- Llama3-70B, PersianMind, and Dorna -- were evaluated using this benchmark. Their average accuracy was below 50%, meaning they provided fully correct answers to fewer than half of the questions. These results indicate that current language models are still far from being able to solve this benchmark |
| title | FarsEval-PKBETS: A new diverse benchmark for evaluating Persian large language models |
| topic | Computation and Language Artificial Intelligence 68T50 I.2.7; E.0 |
| url | https://arxiv.org/abs/2504.14690 |