Déjà Vu: Multilingual LLM Evaluation through the Lens of Machine Translation Evaluation
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866908534069788672 |
|---|---|
| author | Kreutzer, Julia Briakou, Eleftheria Agrawal, Sweta Fadaee, Marzieh Tom, Kocmi |
| author_facet | Kreutzer, Julia Briakou, Eleftheria Agrawal, Sweta Fadaee, Marzieh Tom, Kocmi |
| contents | Generation capabilities and language coverage of multilingual large language models (mLLMs) are advancing rapidly. However, evaluation practices for generative abilities of mLLMs are still lacking comprehensiveness, scientific rigor, and consistent adoption across research labs, which undermines their potential to meaningfully guide mLLM development. We draw parallels with machine translation (MT) evaluation, a field that faced similar challenges and has, over decades, developed transparent reporting standards and reliable evaluations for multilingual generative models. Through targeted experiments across key stages of the generative evaluation pipeline, we demonstrate how best practices from MT evaluation can deepen the understanding of quality differences between models. Additionally, we identify essential components for robust meta-evaluation of mLLMs, ensuring the evaluation methods themselves are rigorously assessed. We distill these insights into a checklist of actionable recommendations for mLLM research and development. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_11829 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Déjà Vu: Multilingual LLM Evaluation through the Lens of Machine Translation Evaluation Kreutzer, Julia Briakou, Eleftheria Agrawal, Sweta Fadaee, Marzieh Tom, Kocmi Computation and Language Artificial Intelligence Generation capabilities and language coverage of multilingual large language models (mLLMs) are advancing rapidly. However, evaluation practices for generative abilities of mLLMs are still lacking comprehensiveness, scientific rigor, and consistent adoption across research labs, which undermines their potential to meaningfully guide mLLM development. We draw parallels with machine translation (MT) evaluation, a field that faced similar challenges and has, over decades, developed transparent reporting standards and reliable evaluations for multilingual generative models. Through targeted experiments across key stages of the generative evaluation pipeline, we demonstrate how best practices from MT evaluation can deepen the understanding of quality differences between models. Additionally, we identify essential components for robust meta-evaluation of mLLMs, ensuring the evaluation methods themselves are rigorously assessed. We distill these insights into a checklist of actionable recommendations for mLLM research and development. |
| title | Déjà Vu: Multilingual LLM Evaluation through the Lens of Machine Translation Evaluation |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2504.11829 |