Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2509.19745 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915510994599936 |
|---|---|
| author | Zhang, Pei Chen, Andong Chen, Xi Yang, Baosong Wong, Derek F. Huang, Fei |
| author_facet | Zhang, Pei Chen, Andong Chen, Xi Yang, Baosong Wong, Derek F. Huang, Fei |
| contents | Large language models (LLMs) have expanded from text to speech, giving rise to Speech Large Models (SLMs) that support recognition, translation, and synthesis. A key challenge is aligning speech and text representations, which becomes harder in multilingual settings. Existing methods often freeze LLM parameters and train encoders on multilingual data, but this forces cross-language convergence and limits performance. We introduce Progressive Alignment Representation Training (PART), a multi-stage and multi-task framework that separates within-language from cross-language alignment. During cross-language training, LLM parameters are dynamically activated, and text-based tasks are later introduced to enhance multilingual understanding. Experiments on CommonVoice 15, Fleurs, Wenetspeech, and CoVoST2 show that PART surpasses conventional approaches, with analysis confirming its ability to balance language-specific distinctions and cross-language generalization. These results demonstrate PART's effectiveness and generality for multilingual speech modality alignment. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_19745 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | PART: Progressive Alignment Representation Training for Multilingual Speech-To-Text with LLMs Zhang, Pei Chen, Andong Chen, Xi Yang, Baosong Wong, Derek F. Huang, Fei Computation and Language Sound Large language models (LLMs) have expanded from text to speech, giving rise to Speech Large Models (SLMs) that support recognition, translation, and synthesis. A key challenge is aligning speech and text representations, which becomes harder in multilingual settings. Existing methods often freeze LLM parameters and train encoders on multilingual data, but this forces cross-language convergence and limits performance. We introduce Progressive Alignment Representation Training (PART), a multi-stage and multi-task framework that separates within-language from cross-language alignment. During cross-language training, LLM parameters are dynamically activated, and text-based tasks are later introduced to enhance multilingual understanding. Experiments on CommonVoice 15, Fleurs, Wenetspeech, and CoVoST2 show that PART surpasses conventional approaches, with analysis confirming its ability to balance language-specific distinctions and cross-language generalization. These results demonstrate PART's effectiveness and generality for multilingual speech modality alignment. |
| title | PART: Progressive Alignment Representation Training for Multilingual Speech-To-Text with LLMs |
| topic | Computation and Language Sound |
| url | https://arxiv.org/abs/2509.19745 |