Cross-lingual Human-Preference Alignment for Neural Machine Translation with Direct Quality Optimization
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866908564219494400 |
|---|---|
| author | Uhlig, Kaden Wuebker, Joern Reinauer, Raphael DeNero, John |
| author_facet | Uhlig, Kaden Wuebker, Joern Reinauer, Raphael DeNero, John |
| contents | Reinforcement Learning from Human Feedback (RLHF) and derivative techniques like Direct Preference Optimization (DPO) are task-alignment algorithms used to repurpose general, foundational models for specific tasks. We show that applying task-alignment to neural machine translation (NMT) addresses an existing task--data mismatch in NMT, leading to improvements across all languages of a multilingual model, even when task-alignment is only applied to a subset of those languages. We do so by introducing Direct Quality Optimization (DQO), a variant of DPO leveraging a pre-trained translation quality estimation model as a proxy for human preferences, and verify the improvements with both automatic metrics and human evaluation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_17673 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Cross-lingual Human-Preference Alignment for Neural Machine Translation with Direct Quality Optimization Uhlig, Kaden Wuebker, Joern Reinauer, Raphael DeNero, John Computation and Language I.2.7 Reinforcement Learning from Human Feedback (RLHF) and derivative techniques like Direct Preference Optimization (DPO) are task-alignment algorithms used to repurpose general, foundational models for specific tasks. We show that applying task-alignment to neural machine translation (NMT) addresses an existing task--data mismatch in NMT, leading to improvements across all languages of a multilingual model, even when task-alignment is only applied to a subset of those languages. We do so by introducing Direct Quality Optimization (DQO), a variant of DPO leveraging a pre-trained translation quality estimation model as a proxy for human preferences, and verify the improvements with both automatic metrics and human evaluation. |
| title | Cross-lingual Human-Preference Alignment for Neural Machine Translation with Direct Quality Optimization |
| topic | Computation and Language I.2.7 |
| url | https://arxiv.org/abs/2409.17673 |