Diffusion Synthesizer for Efficient Multilingual Speech to Speech Translation
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866909224146042880 |
|---|---|
| author | Hirschkind, Nameer Yu, Xiao Nandwana, Mahesh Kumar Liu, Joseph DuBois, Eloi Le, Dao Thiebaut, Nicolas Sinclair, Colin Spence, Kyle Shang, Charles Abrams, Zoe McGuire, Morgan |
| author_facet | Hirschkind, Nameer Yu, Xiao Nandwana, Mahesh Kumar Liu, Joseph DuBois, Eloi Le, Dao Thiebaut, Nicolas Sinclair, Colin Spence, Kyle Shang, Charles Abrams, Zoe McGuire, Morgan |
| contents | We introduce DiffuseST, a low-latency, direct speech-to-speech translation system capable of preserving the input speaker's voice zero-shot while translating from multiple source languages into English. We experiment with the synthesizer component of the architecture, comparing a Tacotron-based synthesizer to a novel diffusion-based synthesizer. We find the diffusion-based synthesizer to improve MOS and PESQ audio quality metrics by 23\% each and speaker similarity by 5\% while maintaining comparable BLEU scores. Despite having more than double the parameter count, the diffusion synthesizer has lower latency, allowing the entire model to run more than 5$\times$ faster than real-time. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_10223 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Diffusion Synthesizer for Efficient Multilingual Speech to Speech Translation Hirschkind, Nameer Yu, Xiao Nandwana, Mahesh Kumar Liu, Joseph DuBois, Eloi Le, Dao Thiebaut, Nicolas Sinclair, Colin Spence, Kyle Shang, Charles Abrams, Zoe McGuire, Morgan Machine Learning Sound Audio and Speech Processing We introduce DiffuseST, a low-latency, direct speech-to-speech translation system capable of preserving the input speaker's voice zero-shot while translating from multiple source languages into English. We experiment with the synthesizer component of the architecture, comparing a Tacotron-based synthesizer to a novel diffusion-based synthesizer. We find the diffusion-based synthesizer to improve MOS and PESQ audio quality metrics by 23\% each and speaker similarity by 5\% while maintaining comparable BLEU scores. Despite having more than double the parameter count, the diffusion synthesizer has lower latency, allowing the entire model to run more than 5$\times$ faster than real-time. |
| title | Diffusion Synthesizer for Efficient Multilingual Speech to Speech Translation |
| topic | Machine Learning Sound Audio and Speech Processing |
| url | https://arxiv.org/abs/2406.10223 |