Converting Anyone's Voice: End-to-End Expressive Voice Conversion with a Conditional Diffusion Model
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866911864595677184 |
|---|---|
| author | Du, Zongyang Lu, Junchen Zhou, Kun Kaushik, Lakshmish Sisman, Berrak |
| author_facet | Du, Zongyang Lu, Junchen Zhou, Kun Kaushik, Lakshmish Sisman, Berrak |
| contents | Expressive voice conversion (VC) conducts speaker identity conversion for emotional speakers by jointly converting speaker identity and emotional style. Emotional style modeling for arbitrary speakers in expressive VC has not been extensively explored. Previous approaches have relied on vocoders for speech reconstruction, which makes speech quality heavily dependent on the performance of vocoders. A major challenge of expressive VC lies in emotion prosody modeling. To address these challenges, this paper proposes a fully end-to-end expressive VC framework based on a conditional denoising diffusion probabilistic model (DDPM). We utilize speech units derived from self-supervised speech models as content conditioning, along with deep features extracted from speech emotion recognition and speaker verification systems to model emotional style and speaker identity. Objective and subjective evaluations show the effectiveness of our framework. Codes and samples are publicly available. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_01730 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Converting Anyone's Voice: End-to-End Expressive Voice Conversion with a Conditional Diffusion Model Du, Zongyang Lu, Junchen Zhou, Kun Kaushik, Lakshmish Sisman, Berrak Audio and Speech Processing Sound Expressive voice conversion (VC) conducts speaker identity conversion for emotional speakers by jointly converting speaker identity and emotional style. Emotional style modeling for arbitrary speakers in expressive VC has not been extensively explored. Previous approaches have relied on vocoders for speech reconstruction, which makes speech quality heavily dependent on the performance of vocoders. A major challenge of expressive VC lies in emotion prosody modeling. To address these challenges, this paper proposes a fully end-to-end expressive VC framework based on a conditional denoising diffusion probabilistic model (DDPM). We utilize speech units derived from self-supervised speech models as content conditioning, along with deep features extracted from speech emotion recognition and speaker verification systems to model emotional style and speaker identity. Objective and subjective evaluations show the effectiveness of our framework. Codes and samples are publicly available. |
| title | Converting Anyone's Voice: End-to-End Expressive Voice Conversion with a Conditional Diffusion Model |
| topic | Audio and Speech Processing Sound |
| url | https://arxiv.org/abs/2405.01730 |