QE4PE: Word-level Quality Estimation for Human Post-Editing
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866917082204995584 |
|---|---|
| author | Sarti, Gabriele Zouhar, Vilém Chrupała, Grzegorz Guerberof-Arenas, Ana Nissim, Malvina Bisazza, Arianna |
| author_facet | Sarti, Gabriele Zouhar, Vilém Chrupała, Grzegorz Guerberof-Arenas, Ana Nissim, Malvina Bisazza, Arianna |
| contents | Word-level quality estimation (QE) methods aim to detect erroneous spans in machine translations, which can direct and facilitate human post-editing. While the accuracy of word-level QE systems has been assessed extensively, their usability and downstream influence on the speed, quality and editing choices of human post-editing remain understudied. In this study, we investigate the impact of word-level QE on machine translation (MT) post-editing in a realistic setting involving 42 professional post-editors across two translation directions. We compare four error-span highlight modalities, including supervised and uncertainty-based word-level QE methods, for identifying potential errors in the outputs of a state-of-the-art neural MT model. Post-editing effort and productivity are estimated from behavioral logs, while quality improvements are assessed by word- and segment-level human annotation. We find that domain, language and editors' speed are critical factors in determining highlights' effectiveness, with modest differences between human-made and automated QE highlights underlining a gap between accuracy and usability in professional workflows. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_03044 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | QE4PE: Word-level Quality Estimation for Human Post-Editing Sarti, Gabriele Zouhar, Vilém Chrupała, Grzegorz Guerberof-Arenas, Ana Nissim, Malvina Bisazza, Arianna Computation and Language Human-Computer Interaction Word-level quality estimation (QE) methods aim to detect erroneous spans in machine translations, which can direct and facilitate human post-editing. While the accuracy of word-level QE systems has been assessed extensively, their usability and downstream influence on the speed, quality and editing choices of human post-editing remain understudied. In this study, we investigate the impact of word-level QE on machine translation (MT) post-editing in a realistic setting involving 42 professional post-editors across two translation directions. We compare four error-span highlight modalities, including supervised and uncertainty-based word-level QE methods, for identifying potential errors in the outputs of a state-of-the-art neural MT model. Post-editing effort and productivity are estimated from behavioral logs, while quality improvements are assessed by word- and segment-level human annotation. We find that domain, language and editors' speed are critical factors in determining highlights' effectiveness, with modest differences between human-made and automated QE highlights underlining a gap between accuracy and usability in professional workflows. |
| title | QE4PE: Word-level Quality Estimation for Human Post-Editing |
| topic | Computation and Language Human-Computer Interaction |
| url | https://arxiv.org/abs/2503.03044 |