Towards Multilingual LLM Evaluation for European Languages
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912075439144960 |
|---|---|
| author | Thellmann, Klaudia Stadler, Bernhard Fromm, Michael Buschhoff, Jasper Schulze Jude, Alex Barth, Fabio Leveling, Johannes Flores-Herr, Nicolas Köhler, Joachim Jäkel, René Ali, Mehdi |
| author_facet | Thellmann, Klaudia Stadler, Bernhard Fromm, Michael Buschhoff, Jasper Schulze Jude, Alex Barth, Fabio Leveling, Johannes Flores-Herr, Nicolas Köhler, Joachim Jäkel, René Ali, Mehdi |
| contents | The rise of Large Language Models (LLMs) has revolutionized natural language processing across numerous languages and tasks. However, evaluating LLM performance in a consistent and meaningful way across multiple European languages remains challenging, especially due to the scarcity of language-parallel multilingual benchmarks. We introduce a multilingual evaluation approach tailored for European languages. We employ translated versions of five widely-used benchmarks to assess the capabilities of 40 LLMs across 21 European languages. Our contributions include examining the effectiveness of translated benchmarks, assessing the impact of different translation services, and offering a multilingual evaluation framework for LLMs that includes newly created datasets: EU20-MMLU, EU20-HellaSwag, EU20-ARC, EU20-TruthfulQA, and EU20-GSM8K. The benchmarks and results are made publicly available to encourage further research in multilingual LLM evaluation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_08928 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Towards Multilingual LLM Evaluation for European Languages Thellmann, Klaudia Stadler, Bernhard Fromm, Michael Buschhoff, Jasper Schulze Jude, Alex Barth, Fabio Leveling, Johannes Flores-Herr, Nicolas Köhler, Joachim Jäkel, René Ali, Mehdi Computation and Language Artificial Intelligence Machine Learning The rise of Large Language Models (LLMs) has revolutionized natural language processing across numerous languages and tasks. However, evaluating LLM performance in a consistent and meaningful way across multiple European languages remains challenging, especially due to the scarcity of language-parallel multilingual benchmarks. We introduce a multilingual evaluation approach tailored for European languages. We employ translated versions of five widely-used benchmarks to assess the capabilities of 40 LLMs across 21 European languages. Our contributions include examining the effectiveness of translated benchmarks, assessing the impact of different translation services, and offering a multilingual evaluation framework for LLMs that includes newly created datasets: EU20-MMLU, EU20-HellaSwag, EU20-ARC, EU20-TruthfulQA, and EU20-GSM8K. The benchmarks and results are made publicly available to encourage further research in multilingual LLM evaluation. |
| title | Towards Multilingual LLM Evaluation for European Languages |
| topic | Computation and Language Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2410.08928 |