Fine-Tuned Machine Translation Metrics Struggle in Unseen Domains
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911901470949376 |
|---|---|
| author | Zouhar, Vilém Ding, Shuoyang Currey, Anna Badeka, Tatyana Wang, Jenyuan Thompson, Brian |
| author_facet | Zouhar, Vilém Ding, Shuoyang Currey, Anna Badeka, Tatyana Wang, Jenyuan Thompson, Brian |
| contents | We introduce a new, extensive multidimensional quality metrics (MQM) annotated dataset covering 11 language pairs in the biomedical domain. We use this dataset to investigate whether machine translation (MT) metrics which are fine-tuned on human-generated MT quality judgements are robust to domain shifts between training and inference. We find that fine-tuned metrics exhibit a substantial performance drop in the unseen domain scenario relative to metrics that rely on the surface form, as well as pre-trained metrics which are not fine-tuned on MT quality judgments. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_18747 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Fine-Tuned Machine Translation Metrics Struggle in Unseen Domains Zouhar, Vilém Ding, Shuoyang Currey, Anna Badeka, Tatyana Wang, Jenyuan Thompson, Brian Computation and Language Artificial Intelligence We introduce a new, extensive multidimensional quality metrics (MQM) annotated dataset covering 11 language pairs in the biomedical domain. We use this dataset to investigate whether machine translation (MT) metrics which are fine-tuned on human-generated MT quality judgements are robust to domain shifts between training and inference. We find that fine-tuned metrics exhibit a substantial performance drop in the unseen domain scenario relative to metrics that rely on the surface form, as well as pre-trained metrics which are not fine-tuned on MT quality judgments. |
| title | Fine-Tuned Machine Translation Metrics Struggle in Unseen Domains |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2402.18747 |