Making LLMs Better Many-to-Many Speech-to-Text Translators with Curriculum Learning
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866913894506692608 |
|---|---|
| author | Du, Yexing Pan, Youcheng Ma, Ziyang Yang, Bo Yang, Yifan Deng, Keqi Chen, Xie Xiang, Yang Liu, Ming Qin, Bing |
| author_facet | Du, Yexing Pan, Youcheng Ma, Ziyang Yang, Bo Yang, Yifan Deng, Keqi Chen, Xie Xiang, Yang Liu, Ming Qin, Bing |
| contents | Multimodal Large Language Models (MLLMs) have achieved significant success in Speech-to-Text Translation (S2TT) tasks. While most existing research has focused on English-centric translation directions, the exploration of many-to-many translation is still limited by the scarcity of parallel data. To address this, we propose a three-stage curriculum learning strategy that leverages the machine translation capabilities of large language models and adapts them to S2TT tasks, enabling effective learning in low-resource settings. We trained MLLMs with varying parameter sizes (3B, 7B, and 32B) and evaluated the proposed strategy using the FLEURS and CoVoST-2 datasets. Experimental results show that the proposed strategy achieves state-of-the-art average performance in $15\times14$ language pairs, requiring fewer than 10 hours of speech data per language to achieve competitive results. The source code and models are released at https://github.com/yxduir/LLM-SRT. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_19510 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Making LLMs Better Many-to-Many Speech-to-Text Translators with Curriculum Learning Du, Yexing Pan, Youcheng Ma, Ziyang Yang, Bo Yang, Yifan Deng, Keqi Chen, Xie Xiang, Yang Liu, Ming Qin, Bing Computation and Language Multimodal Large Language Models (MLLMs) have achieved significant success in Speech-to-Text Translation (S2TT) tasks. While most existing research has focused on English-centric translation directions, the exploration of many-to-many translation is still limited by the scarcity of parallel data. To address this, we propose a three-stage curriculum learning strategy that leverages the machine translation capabilities of large language models and adapts them to S2TT tasks, enabling effective learning in low-resource settings. We trained MLLMs with varying parameter sizes (3B, 7B, and 32B) and evaluated the proposed strategy using the FLEURS and CoVoST-2 datasets. Experimental results show that the proposed strategy achieves state-of-the-art average performance in $15\times14$ language pairs, requiring fewer than 10 hours of speech data per language to achieve competitive results. The source code and models are released at https://github.com/yxduir/LLM-SRT. |
| title | Making LLMs Better Many-to-Many Speech-to-Text Translators with Curriculum Learning |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2409.19510 |