PROST-LLM: Progressively Enhancing the Speech-to-Speech Translation Capability in LLMs
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2026
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866918301839392768 |
|---|---|
| author | Xu, Jing Wang, Jiaqi Tan, Daxin Chen, Xiao |
| author_facet | Xu, Jing Wang, Jiaqi Tan, Daxin Chen, Xiao |
| contents | Although Large Language Models (LLMs) excel in many tasks, their application to Speech-to-Speech Translation (S2ST) is underexplored and hindered by data scarcity. To bridge this gap, we propose PROST-LLM (PROgressive Speech-to-speech Translation) to enhance the S2ST capabilities in LLMs progressively. First, we fine-tune the LLMs with the CVSS corpus, employing designed tri-task learning and chain of modality methods to boost the initial performance. Then, leveraging the fine-tuned model, we generate preference pairs through self-sampling and back-translation without human evaluation. Finally, these preference pairs are used for preference optimization to enhance the model's S2ST capability further. Extensive experiments confirm the effectiveness of our proposed PROST-LLM in improving the S2ST capability of LLMs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_16618 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | PROST-LLM: Progressively Enhancing the Speech-to-Speech Translation Capability in LLMs Xu, Jing Wang, Jiaqi Tan, Daxin Chen, Xiao Computation and Language Although Large Language Models (LLMs) excel in many tasks, their application to Speech-to-Speech Translation (S2ST) is underexplored and hindered by data scarcity. To bridge this gap, we propose PROST-LLM (PROgressive Speech-to-speech Translation) to enhance the S2ST capabilities in LLMs progressively. First, we fine-tune the LLMs with the CVSS corpus, employing designed tri-task learning and chain of modality methods to boost the initial performance. Then, leveraging the fine-tuned model, we generate preference pairs through self-sampling and back-translation without human evaluation. Finally, these preference pairs are used for preference optimization to enhance the model's S2ST capability further. Extensive experiments confirm the effectiveness of our proposed PROST-LLM in improving the S2ST capability of LLMs. |
| title | PROST-LLM: Progressively Enhancing the Speech-to-Speech Translation Capability in LLMs |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2601.16618 |