Textless Streaming Speech-to-Speech Translation using Semantic Speech Tokens
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866914964621492224 |
|---|---|
| author | Zhao, Jinzheng Moritz, Niko Lakomkin, Egor Xie, Ruiming Xiu, Zhiping Zmolikova, Katerina Ahmed, Zeeshan Gaur, Yashesh Le, Duc Fuegen, Christian |
| author_facet | Zhao, Jinzheng Moritz, Niko Lakomkin, Egor Xie, Ruiming Xiu, Zhiping Zmolikova, Katerina Ahmed, Zeeshan Gaur, Yashesh Le, Duc Fuegen, Christian |
| contents | Cascaded speech-to-speech translation systems often suffer from the error accumulation problem and high latency, which is a result of cascaded modules whose inference delays accumulate. In this paper, we propose a transducer-based speech translation model that outputs discrete speech tokens in a low-latency streaming fashion. This approach eliminates the need for generating text output first, followed by machine translation (MT) and text-to-speech (TTS) systems. The produced speech tokens can be directly used to generate a speech signal with low latency by utilizing an acoustic language model (LM) to obtain acoustic tokens and an audio codec model to retrieve the waveform. Experimental results show that the proposed method outperforms other existing approaches and achieves state-of-the-art results for streaming translation in terms of BLEU, average latency, and BLASER 2.0 scores for multiple language pairs using the CVSS-C dataset as a benchmark. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_03298 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Textless Streaming Speech-to-Speech Translation using Semantic Speech Tokens Zhao, Jinzheng Moritz, Niko Lakomkin, Egor Xie, Ruiming Xiu, Zhiping Zmolikova, Katerina Ahmed, Zeeshan Gaur, Yashesh Le, Duc Fuegen, Christian Audio and Speech Processing Cascaded speech-to-speech translation systems often suffer from the error accumulation problem and high latency, which is a result of cascaded modules whose inference delays accumulate. In this paper, we propose a transducer-based speech translation model that outputs discrete speech tokens in a low-latency streaming fashion. This approach eliminates the need for generating text output first, followed by machine translation (MT) and text-to-speech (TTS) systems. The produced speech tokens can be directly used to generate a speech signal with low latency by utilizing an acoustic language model (LM) to obtain acoustic tokens and an audio codec model to retrieve the waveform. Experimental results show that the proposed method outperforms other existing approaches and achieves state-of-the-art results for streaming translation in terms of BLEU, average latency, and BLASER 2.0 scores for multiple language pairs using the CVSS-C dataset as a benchmark. |
| title | Textless Streaming Speech-to-Speech Translation using Semantic Speech Tokens |
| topic | Audio and Speech Processing |
| url | https://arxiv.org/abs/2410.03298 |