Towards Expressive Zero-Shot Speech Synthesis with Hierarchical Prosody Modeling
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866914831615918080 |
|---|---|
| author | Jiang, Yuepeng Li, Tao Yang, Fengyu Xie, Lei Meng, Meng Wang, Yujun |
| author_facet | Jiang, Yuepeng Li, Tao Yang, Fengyu Xie, Lei Meng, Meng Wang, Yujun |
| contents | Recent research in zero-shot speech synthesis has made significant progress in speaker similarity. However, current efforts focus on timbre generalization rather than prosody modeling, which results in limited naturalness and expressiveness. To address this, we introduce a novel speech synthesis model trained on large-scale datasets, including both timbre and hierarchical prosody modeling. As timbre is a global attribute closely linked to expressiveness, we adopt a global vector to model speaker timbre while guiding prosody modeling. Besides, given that prosody contains both global consistency and local variations, we introduce a diffusion model as the pitch predictor and employ a prosody adaptor to model prosody hierarchically, further enhancing the prosody quality of the synthesized speech. Experimental results show that our model not only maintains comparable timbre quality to the baseline but also exhibits better naturalness and expressiveness. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_05681 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Towards Expressive Zero-Shot Speech Synthesis with Hierarchical Prosody Modeling Jiang, Yuepeng Li, Tao Yang, Fengyu Xie, Lei Meng, Meng Wang, Yujun Sound Audio and Speech Processing Recent research in zero-shot speech synthesis has made significant progress in speaker similarity. However, current efforts focus on timbre generalization rather than prosody modeling, which results in limited naturalness and expressiveness. To address this, we introduce a novel speech synthesis model trained on large-scale datasets, including both timbre and hierarchical prosody modeling. As timbre is a global attribute closely linked to expressiveness, we adopt a global vector to model speaker timbre while guiding prosody modeling. Besides, given that prosody contains both global consistency and local variations, we introduce a diffusion model as the pitch predictor and employ a prosody adaptor to model prosody hierarchically, further enhancing the prosody quality of the synthesized speech. Experimental results show that our model not only maintains comparable timbre quality to the baseline but also exhibits better naturalness and expressiveness. |
| title | Towards Expressive Zero-Shot Speech Synthesis with Hierarchical Prosody Modeling |
| topic | Sound Audio and Speech Processing |
| url | https://arxiv.org/abs/2406.05681 |