Investigating the potential of Sparse Mixtures-of-Experts for multi-domain neural machine translation
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866914853399035904 |
|---|---|
| author | Chirkova, Nadezhda Nikoulina, Vassilina Meunier, Jean-Luc Bérard, Alexandre |
| author_facet | Chirkova, Nadezhda Nikoulina, Vassilina Meunier, Jean-Luc Bérard, Alexandre |
| contents | We focus on multi-domain Neural Machine Translation, with the goal of developing efficient models which can handle data from various domains seen during training and are robust to domains unseen during training. We hypothesize that Sparse Mixture-of-Experts (SMoE) models are a good fit for this task, as they enable efficient model scaling, which helps to accommodate a variety of multi-domain data, and allow flexible sharing of parameters between domains, potentially enabling knowledge transfer between similar domains and limiting negative transfer. We conduct a series of experiments aimed at validating the utility of SMoE for the multi-domain scenario, and find that a straightforward width scaling of Transformer is a simpler and surprisingly more efficient approach in practice, and reaches the same performance level as SMoE. We also search for a better recipe for robustness of multi-domain systems, highlighting the importance of mixing-in a generic domain, i.e. Paracrawl, and introducing a simple technique, domain randomization. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_01126 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Investigating the potential of Sparse Mixtures-of-Experts for multi-domain neural machine translation Chirkova, Nadezhda Nikoulina, Vassilina Meunier, Jean-Luc Bérard, Alexandre Computation and Language Artificial Intelligence We focus on multi-domain Neural Machine Translation, with the goal of developing efficient models which can handle data from various domains seen during training and are robust to domains unseen during training. We hypothesize that Sparse Mixture-of-Experts (SMoE) models are a good fit for this task, as they enable efficient model scaling, which helps to accommodate a variety of multi-domain data, and allow flexible sharing of parameters between domains, potentially enabling knowledge transfer between similar domains and limiting negative transfer. We conduct a series of experiments aimed at validating the utility of SMoE for the multi-domain scenario, and find that a straightforward width scaling of Transformer is a simpler and surprisingly more efficient approach in practice, and reaches the same performance level as SMoE. We also search for a better recipe for robustness of multi-domain systems, highlighting the importance of mixing-in a generic domain, i.e. Paracrawl, and introducing a simple technique, domain randomization. |
| title | Investigating the potential of Sparse Mixtures-of-Experts for multi-domain neural machine translation |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2407.01126 |