DiffCSS: Diverse and Expressive Conversational Speech Synthesis with Diffusion Models
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866929734161530880 |
|---|---|
| author | wu, Weihao Lin, Zhiwei Zhou, Yixuan Li, Jingbei Niu, Rui Wu, Qinghua Cao, Songjun Ma, Long Wu, Zhiyong |
| author_facet | wu, Weihao Lin, Zhiwei Zhou, Yixuan Li, Jingbei Niu, Rui Wu, Qinghua Cao, Songjun Ma, Long Wu, Zhiyong |
| contents | Conversational speech synthesis (CSS) aims to synthesize both contextually appropriate and expressive speech, and considerable efforts have been made to enhance the understanding of conversational context. However, existing CSS systems are limited to deterministic prediction, overlooking the diversity of potential responses. Moreover, they rarely employ language model (LM)-based TTS backbones, limiting the naturalness and quality of synthesized speech. To address these issues, in this paper, we propose DiffCSS, an innovative CSS framework that leverages diffusion models and an LM-based TTS backbone to generate diverse, expressive, and contextually coherent speech. A diffusion-based context-aware prosody predictor is proposed to sample diverse prosody embeddings conditioned on multimodal conversational context. Then a prosody-controllable LM-based TTS backbone is developed to synthesize high-quality speech with sampled prosody embeddings. Experimental results demonstrate that the synthesized speech from DiffCSS is more diverse, contextually coherent, and expressive than existing CSS systems |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_19924 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | DiffCSS: Diverse and Expressive Conversational Speech Synthesis with Diffusion Models wu, Weihao Lin, Zhiwei Zhou, Yixuan Li, Jingbei Niu, Rui Wu, Qinghua Cao, Songjun Ma, Long Wu, Zhiyong Sound Artificial Intelligence Audio and Speech Processing Conversational speech synthesis (CSS) aims to synthesize both contextually appropriate and expressive speech, and considerable efforts have been made to enhance the understanding of conversational context. However, existing CSS systems are limited to deterministic prediction, overlooking the diversity of potential responses. Moreover, they rarely employ language model (LM)-based TTS backbones, limiting the naturalness and quality of synthesized speech. To address these issues, in this paper, we propose DiffCSS, an innovative CSS framework that leverages diffusion models and an LM-based TTS backbone to generate diverse, expressive, and contextually coherent speech. A diffusion-based context-aware prosody predictor is proposed to sample diverse prosody embeddings conditioned on multimodal conversational context. Then a prosody-controllable LM-based TTS backbone is developed to synthesize high-quality speech with sampled prosody embeddings. Experimental results demonstrate that the synthesized speech from DiffCSS is more diverse, contextually coherent, and expressive than existing CSS systems |
| title | DiffCSS: Diverse and Expressive Conversational Speech Synthesis with Diffusion Models |
| topic | Sound Artificial Intelligence Audio and Speech Processing |
| url | https://arxiv.org/abs/2502.19924 |