MQM Re-Annotation: A Technique for Collaborative Evaluation of Machine Translation
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866918176171753472 |
|---|---|
| author | Riley, Parker Deutsch, Daniel Finkelstein, Mara DiIanni, Colten Juraska, Juraj Freitag, Markus |
| author_facet | Riley, Parker Deutsch, Daniel Finkelstein, Mara DiIanni, Colten Juraska, Juraj Freitag, Markus |
| contents | Human evaluation of machine translation is in an arms race with translation model quality: as our models get better, our evaluation methods need to be improved to ensure that quality gains are not lost in evaluation noise. To this end, we experiment with a two-stage version of the current state-of-the-art translation evaluation paradigm (MQM), which we call MQM re-annotation. In this setup, an MQM annotator reviews and edits a set of pre-existing MQM annotations, that may have come from themselves, another human annotator, or an automatic MQM annotation system. We demonstrate that rater behavior in re-annotation aligns with our goals, and that re-annotation results in higher-quality annotations, mostly due to finding errors that were missed during the first pass. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_24664 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | MQM Re-Annotation: A Technique for Collaborative Evaluation of Machine Translation Riley, Parker Deutsch, Daniel Finkelstein, Mara DiIanni, Colten Juraska, Juraj Freitag, Markus Computation and Language Human evaluation of machine translation is in an arms race with translation model quality: as our models get better, our evaluation methods need to be improved to ensure that quality gains are not lost in evaluation noise. To this end, we experiment with a two-stage version of the current state-of-the-art translation evaluation paradigm (MQM), which we call MQM re-annotation. In this setup, an MQM annotator reviews and edits a set of pre-existing MQM annotations, that may have come from themselves, another human annotator, or an automatic MQM annotation system. We demonstrate that rater behavior in re-annotation aligns with our goals, and that re-annotation results in higher-quality annotations, mostly due to finding errors that were missed during the first pass. |
| title | MQM Re-Annotation: A Technique for Collaborative Evaluation of Machine Translation |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2510.24664 |