NorEval: A Norwegian Language Understanding and Generation Evaluation Benchmark
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866918045877796864 |
|---|---|
| author | Mikhailov, Vladislav Enstad, Tita Samuel, David Farsethås, Hans Christian Kutuzov, Andrey Velldal, Erik Øvrelid, Lilja |
| author_facet | Mikhailov, Vladislav Enstad, Tita Samuel, David Farsethås, Hans Christian Kutuzov, Andrey Velldal, Erik Øvrelid, Lilja |
| contents | This paper introduces NorEval, a new and comprehensive evaluation suite for large-scale standardized benchmarking of Norwegian generative language models (LMs). NorEval consists of 24 high-quality human-created datasets -- of which five are created from scratch. In contrast to existing benchmarks for Norwegian, NorEval covers a broad spectrum of task categories targeting Norwegian language understanding and generation, establishes human baselines, and focuses on both of the official written standards of the Norwegian language: Bokmål and Nynorsk. All our datasets and a collection of over 100 human-written prompts are integrated into LM Evaluation Harness, ensuring flexible and reproducible evaluation. We describe the NorEval design and present the results of benchmarking 19 open-source pre-trained and instruction-tuned LMs for Norwegian in various scenarios. Our benchmark, evaluation framework, and annotation materials are publicly available. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_07749 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | NorEval: A Norwegian Language Understanding and Generation Evaluation Benchmark Mikhailov, Vladislav Enstad, Tita Samuel, David Farsethås, Hans Christian Kutuzov, Andrey Velldal, Erik Øvrelid, Lilja Computation and Language Artificial Intelligence This paper introduces NorEval, a new and comprehensive evaluation suite for large-scale standardized benchmarking of Norwegian generative language models (LMs). NorEval consists of 24 high-quality human-created datasets -- of which five are created from scratch. In contrast to existing benchmarks for Norwegian, NorEval covers a broad spectrum of task categories targeting Norwegian language understanding and generation, establishes human baselines, and focuses on both of the official written standards of the Norwegian language: Bokmål and Nynorsk. All our datasets and a collection of over 100 human-written prompts are integrated into LM Evaluation Harness, ensuring flexible and reproducible evaluation. We describe the NorEval design and present the results of benchmarking 19 open-source pre-trained and instruction-tuned LMs for Norwegian in various scenarios. Our benchmark, evaluation framework, and annotation materials are publicly available. |
| title | NorEval: A Norwegian Language Understanding and Generation Evaluation Benchmark |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2504.07749 |