EMMeTT: Efficient Multimodal Machine Translation Training
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912037239521280 |
|---|---|
| author | Żelasko, Piotr Chen, Zhehuai Wang, Mengru Galvez, Daniel Hrinchuk, Oleksii Ding, Shuoyang Hu, Ke Balam, Jagadeesh Lavrukhin, Vitaly Ginsburg, Boris |
| author_facet | Żelasko, Piotr Chen, Zhehuai Wang, Mengru Galvez, Daniel Hrinchuk, Oleksii Ding, Shuoyang Hu, Ke Balam, Jagadeesh Lavrukhin, Vitaly Ginsburg, Boris |
| contents | A rising interest in the modality extension of foundation language models warrants discussion on the most effective, and efficient, multimodal training approach. This work focuses on neural machine translation (NMT) and proposes a joint multimodal training regime of Speech-LLM to include automatic speech translation (AST). We investigate two different foundation model architectures, decoder-only GPT and encoder-decoder T5, extended with Canary-1B's speech encoder. To handle joint multimodal training, we propose a novel training framework called EMMeTT. EMMeTT improves training efficiency with the following: balanced sampling across languages, datasets, and modalities; efficient sequential data iteration; and a novel 2D bucketing scheme for multimodal data, complemented by a batch size optimizer (OOMptimizer). We show that a multimodal training consistently helps with both architectures. Moreover, SALM-T5 trained with EMMeTT retains the original NMT capability while outperforming AST baselines on four-language subsets of FLORES and FLEURS. The resultant Multimodal Translation Model produces strong text and speech translation results at the same time. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_13523 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | EMMeTT: Efficient Multimodal Machine Translation Training Żelasko, Piotr Chen, Zhehuai Wang, Mengru Galvez, Daniel Hrinchuk, Oleksii Ding, Shuoyang Hu, Ke Balam, Jagadeesh Lavrukhin, Vitaly Ginsburg, Boris Computation and Language Sound Audio and Speech Processing A rising interest in the modality extension of foundation language models warrants discussion on the most effective, and efficient, multimodal training approach. This work focuses on neural machine translation (NMT) and proposes a joint multimodal training regime of Speech-LLM to include automatic speech translation (AST). We investigate two different foundation model architectures, decoder-only GPT and encoder-decoder T5, extended with Canary-1B's speech encoder. To handle joint multimodal training, we propose a novel training framework called EMMeTT. EMMeTT improves training efficiency with the following: balanced sampling across languages, datasets, and modalities; efficient sequential data iteration; and a novel 2D bucketing scheme for multimodal data, complemented by a batch size optimizer (OOMptimizer). We show that a multimodal training consistently helps with both architectures. Moreover, SALM-T5 trained with EMMeTT retains the original NMT capability while outperforming AST baselines on four-language subsets of FLORES and FLEURS. The resultant Multimodal Translation Model produces strong text and speech translation results at the same time. |
| title | EMMeTT: Efficient Multimodal Machine Translation Training |
| topic | Computation and Language Sound Audio and Speech Processing |
| url | https://arxiv.org/abs/2409.13523 |