Who Benchmarks the Benchmarks? A Case Study of LLM Evaluation in Icelandic
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2026
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866917349915885568 |
|---|---|
| author | Ingimundarson, Finnur Ágúst Friðriksdóttir, Steinunn Rut Ármannsson, Bjarki Nowenstein, Iris Edda Steingrímsson, Steinþór |
| author_facet | Ingimundarson, Finnur Ágúst Friðriksdóttir, Steinunn Rut Ármannsson, Bjarki Nowenstein, Iris Edda Steingrímsson, Steinþór |
| contents | This paper evaluates current Large Language Model (LLM) benchmarking for Icelandic, identifies problems, and calls for improved evaluation methods in low/medium-resource languages in particular. We show that benchmarks that include synthetic or machine-translated data that have not been verified in any way, commonly contain severely flawed test examples that are likely to skew the results and undermine the tests' validity. We warn against the use of such methods without verification in low/medium-resource settings as the translation quality can, at best, only be as good as MT quality for a given language at any given time. Indeed, the results of our quantitative error analysis on existing benchmarks for Icelandic show clear differences between human-authored/-translated benchmarks vs. synthetic or machine-translated benchmarks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_16406 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Who Benchmarks the Benchmarks? A Case Study of LLM Evaluation in Icelandic Ingimundarson, Finnur Ágúst Friðriksdóttir, Steinunn Rut Ármannsson, Bjarki Nowenstein, Iris Edda Steingrímsson, Steinþór Computation and Language Artificial Intelligence This paper evaluates current Large Language Model (LLM) benchmarking for Icelandic, identifies problems, and calls for improved evaluation methods in low/medium-resource languages in particular. We show that benchmarks that include synthetic or machine-translated data that have not been verified in any way, commonly contain severely flawed test examples that are likely to skew the results and undermine the tests' validity. We warn against the use of such methods without verification in low/medium-resource settings as the translation quality can, at best, only be as good as MT quality for a given language at any given time. Indeed, the results of our quantitative error analysis on existing benchmarks for Icelandic show clear differences between human-authored/-translated benchmarks vs. synthetic or machine-translated benchmarks. |
| title | Who Benchmarks the Benchmarks? A Case Study of LLM Evaluation in Icelandic |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2603.16406 |