Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2510.24450 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908627952992256 |
|---|---|
| author | Vintar, Špela Pungeršek, Taja Kuzman Brglez, Mojca Ljubešić, Nikola |
| author_facet | Vintar, Špela Pungeršek, Taja Kuzman Brglez, Mojca Ljubešić, Nikola |
| contents | While new benchmarks for large language models (LLMs) are being developed continuously to catch up with the growing capabilities of new models and AI in general, using and evaluating LLMs in non-English languages remains a little-charted landscape. We give a concise overview of recent developments in LLM benchmarking, and then propose a new taxonomy for the categorization of benchmarks that is tailored to multilingual or non-English use scenarios. We further propose a set of best practices and quality standards that could lead to a more coordinated development of benchmarks for European languages. Among other recommendations, we advocate for a higher language and culture sensitivity of evaluation methods. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_24450 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Charting the European LLM Benchmarking Landscape: A New Taxonomy and a Set of Best Practices Vintar, Špela Pungeršek, Taja Kuzman Brglez, Mojca Ljubešić, Nikola Computation and Language Artificial Intelligence While new benchmarks for large language models (LLMs) are being developed continuously to catch up with the growing capabilities of new models and AI in general, using and evaluating LLMs in non-English languages remains a little-charted landscape. We give a concise overview of recent developments in LLM benchmarking, and then propose a new taxonomy for the categorization of benchmarks that is tailored to multilingual or non-English use scenarios. We further propose a set of best practices and quality standards that could lead to a more coordinated development of benchmarks for European languages. Among other recommendations, we advocate for a higher language and culture sensitivity of evaluation methods. |
| title | Charting the European LLM Benchmarking Landscape: A New Taxonomy and a Set of Best Practices |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2510.24450 |