Can the Variation of Model Weights be used as a Criterion for Self-Paced Multilingual NMT?
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910634853007360 |
|---|---|
| author | Atrio, Àlex R. Allemann, Alexis Dolamic, Ljiljana Popescu-Belis, Andrei |
| author_facet | Atrio, Àlex R. Allemann, Alexis Dolamic, Ljiljana Popescu-Belis, Andrei |
| contents | Many-to-one neural machine translation systems improve over one-to-one systems when training data is scarce. In this paper, we design and test a novel algorithm for selecting the language of minibatches when training such systems. The algorithm changes the language of the minibatch when the weights of the model do not evolve significantly, as measured by the smoothed KL divergence between all layers of the Transformer network. This algorithm outperforms the use of alternating monolingual batches, but not the use of shuffled batches, in terms of translation quality (measured with BLEU and COMET) and convergence speed. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_04147 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Can the Variation of Model Weights be used as a Criterion for Self-Paced Multilingual NMT? Atrio, Àlex R. Allemann, Alexis Dolamic, Ljiljana Popescu-Belis, Andrei Computation and Language Many-to-one neural machine translation systems improve over one-to-one systems when training data is scarce. In this paper, we design and test a novel algorithm for selecting the language of minibatches when training such systems. The algorithm changes the language of the minibatch when the weights of the model do not evolve significantly, as measured by the smoothed KL divergence between all layers of the Transformer network. This algorithm outperforms the use of alternating monolingual batches, but not the use of shuffled batches, in terms of translation quality (measured with BLEU and COMET) and convergence speed. |
| title | Can the Variation of Model Weights be used as a Criterion for Self-Paced Multilingual NMT? |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2410.04147 |