Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2410.11056 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912073210920960 |
|---|---|
| author | Liu, Zhongtao Riley, Parker Deutsch, Daniel Lui, Alison Niu, Mengmeng Shah, Apu Freitag, Markus |
| author_facet | Liu, Zhongtao Riley, Parker Deutsch, Daniel Lui, Alison Niu, Mengmeng Shah, Apu Freitag, Markus |
| contents | Collecting high-quality translations is crucial for the development and evaluation of machine translation systems. However, traditional human-only approaches are costly and slow. This study presents a comprehensive investigation of 11 approaches for acquiring translation data, including human-only, machineonly, and hybrid approaches. Our findings demonstrate that human-machine collaboration can match or even exceed the quality of human-only translations, while being more cost-efficient. Error analysis reveals the complementary strengths between human and machine contributions, highlighting the effectiveness of collaborative methods. Cost analysis further demonstrates the economic benefits of human-machine collaboration methods, with some approaches achieving top-tier quality at around 60% of the cost of traditional methods. We release a publicly available dataset containing nearly 18,000 segments of varying translation quality with corresponding human ratings to facilitate future research. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_11056 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Beyond Human-Only: Evaluating Human-Machine Collaboration for Collecting High-Quality Translation Data Liu, Zhongtao Riley, Parker Deutsch, Daniel Lui, Alison Niu, Mengmeng Shah, Apu Freitag, Markus Computation and Language Collecting high-quality translations is crucial for the development and evaluation of machine translation systems. However, traditional human-only approaches are costly and slow. This study presents a comprehensive investigation of 11 approaches for acquiring translation data, including human-only, machineonly, and hybrid approaches. Our findings demonstrate that human-machine collaboration can match or even exceed the quality of human-only translations, while being more cost-efficient. Error analysis reveals the complementary strengths between human and machine contributions, highlighting the effectiveness of collaborative methods. Cost analysis further demonstrates the economic benefits of human-machine collaboration methods, with some approaches achieving top-tier quality at around 60% of the cost of traditional methods. We release a publicly available dataset containing nearly 18,000 segments of varying translation quality with corresponding human ratings to facilitate future research. |
| title | Beyond Human-Only: Evaluating Human-Machine Collaboration for Collecting High-Quality Translation Data |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2410.11056 |