Quantifying the Impact of Translation Errors on Multilingual LLM Evaluation
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2026
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866918520076369920 |
|---|---|
| author | Thellmann, Klaudia-Doris Stadler, Bernhard Färber, Michael Lehmann, Jens |
| author_facet | Thellmann, Klaudia-Doris Stadler, Bernhard Färber, Michael Lehmann, Jens |
| contents | Machine-translated benchmarks are widely used to assess the multilingual capabilities of large language models (LLMs), yet translation errors in these benchmarks remain underexplored, raising concerns about the reliability and comparability of multilingual evaluation. We address two practical gaps: (i) how well automatic MQM-style error spans from LLM judges and a span-aware QE baseline (xCOMET-XXL) match expert human span annotations on benchmark translations, and (ii) how strongly translation errors (as opposed to source-side issues in the English original) explain accuracy drops on translated benchmarks. We find that span agreement is non-trivial on naturally occurring benchmark translations, and that target-side translation errors are consistently associated with measurable, percentage-point drops in translated accuracy even after controlling for English correctness and source-side anomalies. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_24904 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Quantifying the Impact of Translation Errors on Multilingual LLM Evaluation Thellmann, Klaudia-Doris Stadler, Bernhard Färber, Michael Lehmann, Jens Computation and Language Machine-translated benchmarks are widely used to assess the multilingual capabilities of large language models (LLMs), yet translation errors in these benchmarks remain underexplored, raising concerns about the reliability and comparability of multilingual evaluation. We address two practical gaps: (i) how well automatic MQM-style error spans from LLM judges and a span-aware QE baseline (xCOMET-XXL) match expert human span annotations on benchmark translations, and (ii) how strongly translation errors (as opposed to source-side issues in the English original) explain accuracy drops on translated benchmarks. We find that span agreement is non-trivial on naturally occurring benchmark translations, and that target-side translation errors are consistently associated with measurable, percentage-point drops in translated accuracy even after controlling for English correctness and source-side anomalies. |
| title | Quantifying the Impact of Translation Errors on Multilingual LLM Evaluation |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2605.24904 |