Multi-Hypothesis Distillation of Multilingual Neural Translation Models for Low-Resource Languages
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912511369936896 |
|---|---|
| author | Galiano-Jiménez, Aarón Pérez-Ortiz, Juan Antonio Sánchez-Martínez, Felipe Sánchez-Cartagena, Víctor M. |
| author_facet | Galiano-Jiménez, Aarón Pérez-Ortiz, Juan Antonio Sánchez-Martínez, Felipe Sánchez-Cartagena, Víctor M. |
| contents | This paper explores sequence-level knowledge distillation (KD) of multilingual pre-trained encoder-decoder translation models. We argue that the teacher model's output distribution holds valuable insights for the student, beyond the approximated mode obtained through beam search (the standard decoding method), and present Multi-Hypothesis Distillation (MHD), a sequence-level KD method that generates multiple translations for each source sentence. This provides a larger representation of the teacher model distribution and exposes the student model to a wider range of target-side prefixes. We leverage $n$-best lists from beam search to guide the student's learning and examine alternative decoding methods to address issues like low variability and the under-representation of infrequent tokens. For low-resource languages, our research shows that while sampling methods may slightly compromise translation quality compared to beam search based approaches, they enhance the generated corpora with greater variability and lexical richness. This ultimately improves student model performance and mitigates the gender bias amplification often associated with KD. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_21568 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Multi-Hypothesis Distillation of Multilingual Neural Translation Models for Low-Resource Languages Galiano-Jiménez, Aarón Pérez-Ortiz, Juan Antonio Sánchez-Martínez, Felipe Sánchez-Cartagena, Víctor M. Computation and Language This paper explores sequence-level knowledge distillation (KD) of multilingual pre-trained encoder-decoder translation models. We argue that the teacher model's output distribution holds valuable insights for the student, beyond the approximated mode obtained through beam search (the standard decoding method), and present Multi-Hypothesis Distillation (MHD), a sequence-level KD method that generates multiple translations for each source sentence. This provides a larger representation of the teacher model distribution and exposes the student model to a wider range of target-side prefixes. We leverage $n$-best lists from beam search to guide the student's learning and examine alternative decoding methods to address issues like low variability and the under-representation of infrequent tokens. For low-resource languages, our research shows that while sampling methods may slightly compromise translation quality compared to beam search based approaches, they enhance the generated corpora with greater variability and lexical richness. This ultimately improves student model performance and mitigates the gender bias amplification often associated with KD. |
| title | Multi-Hypothesis Distillation of Multilingual Neural Translation Models for Low-Resource Languages |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2507.21568 |