RosettaSpeech: Zero-Shot Speech-to-Speech Translation without Parallel Speech
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910022798147584 |
|---|---|
| author | Zheng, Zhisheng Sun, Xiaohang Dinh, Tuan Yanamandra, Abhishek Jain, Abhinav Liu, Zhu Hadap, Sunil Bhat, Vimal Aggarwal, Manoj Medioni, Gerard Harwath, David |
| author_facet | Zheng, Zhisheng Sun, Xiaohang Dinh, Tuan Yanamandra, Abhishek Jain, Abhinav Liu, Zhu Hadap, Sunil Bhat, Vimal Aggarwal, Manoj Medioni, Gerard Harwath, David |
| contents | End-to-end speech-to-speech translation (S2ST) systems typically struggle with a critical data bottleneck: the scarcity of parallel speech-to-speech corpora. To overcome this, we introduce RosettaSpeech, a novel zero-shot framework trained exclusively on monolingual speech-text data augmented by machine translation supervision. Unlike prior works that rely on complex cascaded pseudo-labeling, our approach strategically utilizes text as a semantic bridge during training to synthesize translation targets, thereby eliminating the need for parallel speech pairs while maintaining a direct, end-to-end inference pipeline. Empirical evaluations on the CVSS-C benchmark demonstrate that RosettaSpeech achieves state-of-the-art zero-shot performance, surpassing leading baselines by significant margins - achieving ASR-BLEU scores of 25.17 for German-to-English (+27% relative gain) and 29.86 for Spanish-to-English (+14%). Crucially, our model effectively preserves the source speaker's voice without ever seeing paired speech data. We further analyze the impact of data scaling and demonstrate the model's capability in many-to-one translation, offering a scalable solution for extending high-quality S2ST to "text-rich, speech-poor" languages. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_20974 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | RosettaSpeech: Zero-Shot Speech-to-Speech Translation without Parallel Speech Zheng, Zhisheng Sun, Xiaohang Dinh, Tuan Yanamandra, Abhishek Jain, Abhinav Liu, Zhu Hadap, Sunil Bhat, Vimal Aggarwal, Manoj Medioni, Gerard Harwath, David Audio and Speech Processing Computation and Language Machine Learning End-to-end speech-to-speech translation (S2ST) systems typically struggle with a critical data bottleneck: the scarcity of parallel speech-to-speech corpora. To overcome this, we introduce RosettaSpeech, a novel zero-shot framework trained exclusively on monolingual speech-text data augmented by machine translation supervision. Unlike prior works that rely on complex cascaded pseudo-labeling, our approach strategically utilizes text as a semantic bridge during training to synthesize translation targets, thereby eliminating the need for parallel speech pairs while maintaining a direct, end-to-end inference pipeline. Empirical evaluations on the CVSS-C benchmark demonstrate that RosettaSpeech achieves state-of-the-art zero-shot performance, surpassing leading baselines by significant margins - achieving ASR-BLEU scores of 25.17 for German-to-English (+27% relative gain) and 29.86 for Spanish-to-English (+14%). Crucially, our model effectively preserves the source speaker's voice without ever seeing paired speech data. We further analyze the impact of data scaling and demonstrate the model's capability in many-to-one translation, offering a scalable solution for extending high-quality S2ST to "text-rich, speech-poor" languages. |
| title | RosettaSpeech: Zero-Shot Speech-to-Speech Translation without Parallel Speech |
| topic | Audio and Speech Processing Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2511.20974 |