MCAT: Scaling Many-to-Many Speech-to-Text Translation with MLLMs to 70 Languages
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866917402573275136 |
|---|---|
| author | Du, Yexing Liu, Kaiyuan Pan, Youcheng Yang, Bo Deng, Keqi Chen, Xie Xiang, Yang Liu, Ming Qin, Bing Wang, YaoWei |
| author_facet | Du, Yexing Liu, Kaiyuan Pan, Youcheng Yang, Bo Deng, Keqi Chen, Xie Xiang, Yang Liu, Ming Qin, Bing Wang, YaoWei |
| contents | Multimodal Large Language Models (MLLMs) have achieved great success in Speech-to-Text Translation (S2TT) tasks. However, current research is constrained by two key challenges: language coverage and efficiency. Most of the popular S2TT datasets are substantially English-centric, which restricts the scaling-up of MLLMs' many-to-many translation capabilities. Moreover, the inference speed of MLLMs degrades dramatically when the speech is converted into long sequences (e.g., 750 tokens). To address these limitations, we propose a Multilingual Cost-effective Accelerated Speech-to-Text Translator (MCAT) framework, which includes two innovations. First, a language scaling method that leverages curriculum learning and a data balancing strategy is introduced to extend the language coverage supported by MLLMs to 70 languages and achieve mutual translation among these languages. Second, an optimized speech adapter module is designed to reduce the length of the speech sequence to only 30 tokens. Extensive experiments were conducted on MLLMs of different scales (9B and 27B). The experimental results demonstrate that MCAT not only surpasses state-of-the-art end-to-end models on the FLEURS dataset across 70x69 directions but also enhances inference efficiency. The code and models are released at https://github.com/yxduir/m2m-70. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_01512 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | MCAT: Scaling Many-to-Many Speech-to-Text Translation with MLLMs to 70 Languages Du, Yexing Liu, Kaiyuan Pan, Youcheng Yang, Bo Deng, Keqi Chen, Xie Xiang, Yang Liu, Ming Qin, Bing Wang, YaoWei Computation and Language Multimodal Large Language Models (MLLMs) have achieved great success in Speech-to-Text Translation (S2TT) tasks. However, current research is constrained by two key challenges: language coverage and efficiency. Most of the popular S2TT datasets are substantially English-centric, which restricts the scaling-up of MLLMs' many-to-many translation capabilities. Moreover, the inference speed of MLLMs degrades dramatically when the speech is converted into long sequences (e.g., 750 tokens). To address these limitations, we propose a Multilingual Cost-effective Accelerated Speech-to-Text Translator (MCAT) framework, which includes two innovations. First, a language scaling method that leverages curriculum learning and a data balancing strategy is introduced to extend the language coverage supported by MLLMs to 70 languages and achieve mutual translation among these languages. Second, an optimized speech adapter module is designed to reduce the length of the speech sequence to only 30 tokens. Extensive experiments were conducted on MLLMs of different scales (9B and 27B). The experimental results demonstrate that MCAT not only surpasses state-of-the-art end-to-end models on the FLEURS dataset across 70x69 directions but also enhances inference efficiency. The code and models are released at https://github.com/yxduir/m2m-70. |
| title | MCAT: Scaling Many-to-Many Speech-to-Text Translation with MLLMs to 70 Languages |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2512.01512 |