Saved in:
| Main Authors: | , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2504.04155 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914079957843968 |
|---|---|
| author | Luo, Hengyu Li, Zihao Attieh, Joseph Devkota, Sawal de Gibert, Ona Huang, Xu Ji, Shaoxiong Lin, Peiqin Mantina, Bhavani Sai Praneeth Varma Sreenidhi, Ananda Vázquez, Raúl Wang, Mengjie Yusofi, Samea Yuan, Fei Tiedemann, Jörg |
| author_facet | Luo, Hengyu Li, Zihao Attieh, Joseph Devkota, Sawal de Gibert, Ona Huang, Xu Ji, Shaoxiong Lin, Peiqin Mantina, Bhavani Sai Praneeth Varma Sreenidhi, Ananda Vázquez, Raúl Wang, Mengjie Yusofi, Samea Yuan, Fei Tiedemann, Jörg |
| contents | Large language models (LLMs) are advancing at an unprecedented pace globally, with regions increasingly adopting these models for applications in their primary language. Evaluation of these models in diverse linguistic environments, especially in low-resource languages, has become a major challenge for academia and industry. Existing evaluation frameworks are disproportionately focused on English and a handful of high-resource languages, thereby overlooking the realistic performance of LLMs in multilingual and lower-resource scenarios. To address this gap, we introduce GlotEval, a lightweight framework designed for massively multilingual evaluation. Supporting seven key tasks (machine translation, text classification, summarization, open-ended generation, reading comprehension, sequence labeling, and intrinsic evaluation), spanning over dozens to hundreds of languages, GlotEval highlights consistent multilingual benchmarking, language-specific prompt templates, and non-English-centric machine translation. This enables a precise diagnosis of model strengths and weaknesses in diverse linguistic contexts. A multilingual translation case study demonstrates GlotEval's applicability for multilingual and language-specific evaluations. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_04155 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | GlotEval: A Test Suite for Massively Multilingual Evaluation of Large Language Models Luo, Hengyu Li, Zihao Attieh, Joseph Devkota, Sawal de Gibert, Ona Huang, Xu Ji, Shaoxiong Lin, Peiqin Mantina, Bhavani Sai Praneeth Varma Sreenidhi, Ananda Vázquez, Raúl Wang, Mengjie Yusofi, Samea Yuan, Fei Tiedemann, Jörg Computation and Language Large language models (LLMs) are advancing at an unprecedented pace globally, with regions increasingly adopting these models for applications in their primary language. Evaluation of these models in diverse linguistic environments, especially in low-resource languages, has become a major challenge for academia and industry. Existing evaluation frameworks are disproportionately focused on English and a handful of high-resource languages, thereby overlooking the realistic performance of LLMs in multilingual and lower-resource scenarios. To address this gap, we introduce GlotEval, a lightweight framework designed for massively multilingual evaluation. Supporting seven key tasks (machine translation, text classification, summarization, open-ended generation, reading comprehension, sequence labeling, and intrinsic evaluation), spanning over dozens to hundreds of languages, GlotEval highlights consistent multilingual benchmarking, language-specific prompt templates, and non-English-centric machine translation. This enables a precise diagnosis of model strengths and weaknesses in diverse linguistic contexts. A multilingual translation case study demonstrates GlotEval's applicability for multilingual and language-specific evaluations. |
| title | GlotEval: A Test Suite for Massively Multilingual Evaluation of Large Language Models |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2504.04155 |