EBBS: An Ensemble with Bi-Level Beam Search for Zero-Shot Machine Translation
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915140551573504 |
|---|---|
| author | Wen, Yuqiao Shayegh, Behzad Huang, Chenyang Cao, Yanshuai Mou, Lili |
| author_facet | Wen, Yuqiao Shayegh, Behzad Huang, Chenyang Cao, Yanshuai Mou, Lili |
| contents | The ability of zero-shot translation emerges when we train a multilingual model with certain translation directions; the model can then directly translate in unseen directions. Alternatively, zero-shot translation can be accomplished by pivoting through a third language (e.g., English). In our work, we observe that both direct and pivot translations are noisy and achieve less satisfactory performance. We propose EBBS, an ensemble method with a novel bi-level beam search algorithm, where each ensemble component explores its own prediction step by step at the lower level but they are synchronized by a "soft voting" mechanism at the upper level. Results on two popular multilingual translation datasets show that EBBS consistently outperforms direct and pivot translations as well as existing ensemble techniques. Further, we can distill the ensemble's knowledge back to the multilingual model to improve inference efficiency; profoundly, our EBBS-based distillation does not sacrifice, or even improves, the translation quality. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_00144 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | EBBS: An Ensemble with Bi-Level Beam Search for Zero-Shot Machine Translation Wen, Yuqiao Shayegh, Behzad Huang, Chenyang Cao, Yanshuai Mou, Lili Computation and Language Artificial Intelligence Machine Learning I.2.7; I.2.6; I.2.m; I.5.1; I.7.m The ability of zero-shot translation emerges when we train a multilingual model with certain translation directions; the model can then directly translate in unseen directions. Alternatively, zero-shot translation can be accomplished by pivoting through a third language (e.g., English). In our work, we observe that both direct and pivot translations are noisy and achieve less satisfactory performance. We propose EBBS, an ensemble method with a novel bi-level beam search algorithm, where each ensemble component explores its own prediction step by step at the lower level but they are synchronized by a "soft voting" mechanism at the upper level. Results on two popular multilingual translation datasets show that EBBS consistently outperforms direct and pivot translations as well as existing ensemble techniques. Further, we can distill the ensemble's knowledge back to the multilingual model to improve inference efficiency; profoundly, our EBBS-based distillation does not sacrifice, or even improves, the translation quality. |
| title | EBBS: An Ensemble with Bi-Level Beam Search for Zero-Shot Machine Translation |
| topic | Computation and Language Artificial Intelligence Machine Learning I.2.7; I.2.6; I.2.m; I.5.1; I.7.m |
| url | https://arxiv.org/abs/2403.00144 |