GreekMMLU: A Native-Sourced Multitask Benchmark for Evaluating Language Models in Greek
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2026
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866912986247987200 |
|---|---|
| author | Zhang, Yang Konomi, Mersin Xypolopoulos, Christos Divriotis, Konstantinos Skianis, Konstantinos Nikolentzos, Giannis Stamou, Giorgos Shang, Guokan Vazirgiannis, Michalis |
| author_facet | Zhang, Yang Konomi, Mersin Xypolopoulos, Christos Divriotis, Konstantinos Skianis, Konstantinos Nikolentzos, Giannis Stamou, Giorgos Shang, Guokan Vazirgiannis, Michalis |
| contents | Large Language Models (LLMs) are commonly trained on multilingual corpora that include Greek, yet reliable evaluation benchmarks for Greek-particularly those based on authentic, native-sourced content-remain limited. Existing datasets are often machine-translated from English, failing to capture Greek linguistic and cultural characteristics. We introduce GreekMMLU, a native-sourced benchmark for massive multitask language understanding in Greek, comprising 21,805 multiple-choice questions across 45 subject areas, organized under a newly defined subject taxonomy and annotated with educational difficulty levels spanning primary to professional examinations. All questions are sourced or authored in Greek from academic, professional, and governmental exams. We publicly release 16,857 samples and reserve 4,948 samples for a private leaderboard to enable robust and contamination-resistant evaluation. Evaluations of over 80 open- and closed-source LLMs reveal substantial performance gaps between frontier and open-weight models, as well as between Greek-adapted models and general multilingual ones. Finally, we provide a systematic analysis of factors influencing performance-including model scale, adaptation, and prompting-and derive insights for improving LLM capabilities in Greek. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_05150 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | GreekMMLU: A Native-Sourced Multitask Benchmark for Evaluating Language Models in Greek Zhang, Yang Konomi, Mersin Xypolopoulos, Christos Divriotis, Konstantinos Skianis, Konstantinos Nikolentzos, Giannis Stamou, Giorgos Shang, Guokan Vazirgiannis, Michalis Computation and Language Large Language Models (LLMs) are commonly trained on multilingual corpora that include Greek, yet reliable evaluation benchmarks for Greek-particularly those based on authentic, native-sourced content-remain limited. Existing datasets are often machine-translated from English, failing to capture Greek linguistic and cultural characteristics. We introduce GreekMMLU, a native-sourced benchmark for massive multitask language understanding in Greek, comprising 21,805 multiple-choice questions across 45 subject areas, organized under a newly defined subject taxonomy and annotated with educational difficulty levels spanning primary to professional examinations. All questions are sourced or authored in Greek from academic, professional, and governmental exams. We publicly release 16,857 samples and reserve 4,948 samples for a private leaderboard to enable robust and contamination-resistant evaluation. Evaluations of over 80 open- and closed-source LLMs reveal substantial performance gaps between frontier and open-weight models, as well as between Greek-adapted models and general multilingual ones. Finally, we provide a systematic analysis of factors influencing performance-including model scale, adaptation, and prompting-and derive insights for improving LLM capabilities in Greek. |
| title | GreekMMLU: A Native-Sourced Multitask Benchmark for Evaluating Language Models in Greek |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2602.05150 |