CrossSpeech++: Cross-lingual Speech Synthesis with Decoupled Language and Speaker Generation
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866915083732385792 |
|---|---|
| author | Kim, Ji-Hoon Yang, Hong-Sun Ju, Yoon-Cheol Kim, Il-Hwan Kim, Byeong-Yeol Chung, Joon Son |
| author_facet | Kim, Ji-Hoon Yang, Hong-Sun Ju, Yoon-Cheol Kim, Il-Hwan Kim, Byeong-Yeol Chung, Joon Son |
| contents | The goal of this work is to generate natural speech in multiple languages while maintaining the same speaker identity, a task known as cross-lingual speech synthesis. A key challenge of cross-lingual speech synthesis is the language-speaker entanglement problem, which causes the quality of cross-lingual systems to lag behind that of intra-lingual systems. In this paper, we propose CrossSpeech++, which effectively disentangles language and speaker information and significantly improves the quality of cross-lingual speech synthesis. To this end, we break the complex speech generation pipeline into two simple components: language-dependent and speaker-dependent generators. The language-dependent generator produces linguistic variations that are not biased by specific speaker attributes. The speaker-dependent generator models acoustic variations that characterize speaker identity. By handling each type of information in separate modules, our method can effectively disentangle language and speaker representation. We conduct extensive experiments using various metrics, and demonstrate that CrossSpeech++ achieves significant improvements in cross-lingual speech synthesis, outperforming existing methods by a large margin. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_20048 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | CrossSpeech++: Cross-lingual Speech Synthesis with Decoupled Language and Speaker Generation Kim, Ji-Hoon Yang, Hong-Sun Ju, Yoon-Cheol Kim, Il-Hwan Kim, Byeong-Yeol Chung, Joon Son Audio and Speech Processing Artificial Intelligence Sound Signal Processing The goal of this work is to generate natural speech in multiple languages while maintaining the same speaker identity, a task known as cross-lingual speech synthesis. A key challenge of cross-lingual speech synthesis is the language-speaker entanglement problem, which causes the quality of cross-lingual systems to lag behind that of intra-lingual systems. In this paper, we propose CrossSpeech++, which effectively disentangles language and speaker information and significantly improves the quality of cross-lingual speech synthesis. To this end, we break the complex speech generation pipeline into two simple components: language-dependent and speaker-dependent generators. The language-dependent generator produces linguistic variations that are not biased by specific speaker attributes. The speaker-dependent generator models acoustic variations that characterize speaker identity. By handling each type of information in separate modules, our method can effectively disentangle language and speaker representation. We conduct extensive experiments using various metrics, and demonstrate that CrossSpeech++ achieves significant improvements in cross-lingual speech synthesis, outperforming existing methods by a large margin. |
| title | CrossSpeech++: Cross-lingual Speech Synthesis with Decoupled Language and Speaker Generation |
| topic | Audio and Speech Processing Artificial Intelligence Sound Signal Processing |
| url | https://arxiv.org/abs/2412.20048 |