Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2411.05986 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917096685830144 |
|---|---|
| author | Ramos, Miguel Moura Almeida, Tomás Vareta, Daniel Azevedo, Filipe Agrawal, Sweta Fernandes, Patrick Martins, André F. T. |
| author_facet | Ramos, Miguel Moura Almeida, Tomás Vareta, Daniel Azevedo, Filipe Agrawal, Sweta Fernandes, Patrick Martins, André F. T. |
| contents | Reinforcement learning (RL) has been proven to be an effective and robust method for training neural machine translation systems, especially when paired with powerful reward models that accurately assess translation quality. However, most research has focused on RL methods that use sentence-level feedback, leading to inefficient learning signals due to the reward sparsity problem -- the model receives a single score for the entire sentence. To address this, we propose a novel approach that leverages fine-grained, token-level quality assessments along with error severity levels using RL methods. Specifically, we use xCOMET, a state-of-the-art quality estimation system, as our token-level reward model. We conduct experiments on small and large translation datasets with standard encoder-decoder and large language models-based machine translation systems, comparing the impact of sentence-level versus fine-grained reward signals on translation quality. Our results show that training with token-level rewards improves translation quality across language pairs over baselines according to both automatic and human evaluation. Furthermore, token-level reward optimization improves training stability, evidenced by a steady increase in mean rewards over training epochs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_05986 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Fine-Grained Reward Optimization for Machine Translation using Error Severity Mappings Ramos, Miguel Moura Almeida, Tomás Vareta, Daniel Azevedo, Filipe Agrawal, Sweta Fernandes, Patrick Martins, André F. T. Computation and Language Reinforcement learning (RL) has been proven to be an effective and robust method for training neural machine translation systems, especially when paired with powerful reward models that accurately assess translation quality. However, most research has focused on RL methods that use sentence-level feedback, leading to inefficient learning signals due to the reward sparsity problem -- the model receives a single score for the entire sentence. To address this, we propose a novel approach that leverages fine-grained, token-level quality assessments along with error severity levels using RL methods. Specifically, we use xCOMET, a state-of-the-art quality estimation system, as our token-level reward model. We conduct experiments on small and large translation datasets with standard encoder-decoder and large language models-based machine translation systems, comparing the impact of sentence-level versus fine-grained reward signals on translation quality. Our results show that training with token-level rewards improves translation quality across language pairs over baselines according to both automatic and human evaluation. Furthermore, token-level reward optimization improves training stability, evidenced by a steady increase in mean rewards over training epochs. |
| title | Fine-Grained Reward Optimization for Machine Translation using Error Severity Mappings |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2411.05986 |