DMP-TTS: Disentangled multi-modal Prompting for Controllable Text-to-Speech with Chained Guidance
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912756792295424 |
|---|---|
| author | Yin, Kang Qiang, Chunyu Zhao, Sirui Wang, Xiaopeng Liang, Yuzhe Cai, Pengfei Xu, Tong Zhang, Chen Chen, Enhong |
| author_facet | Yin, Kang Qiang, Chunyu Zhao, Sirui Wang, Xiaopeng Liang, Yuzhe Cai, Pengfei Xu, Tong Zhang, Chen Chen, Enhong |
| contents | Controllable text-to-speech (TTS) systems face significant challenges in achieving independent manipulation of speaker timbre and speaking style, often suffering from entanglement between these attributes. We present DMP-TTS, a latent Diffusion Transformer (DiT) framework with explicit disentanglement and multi-modal prompting. A CLAP-based style encoder (Style-CLAP) aligns cues from reference audio and descriptive text in a shared space and is trained with contrastive learning plus multi-task supervision on style attributes. For fine-grained control during inference, we introduce chained classifier-free guidance (cCFG) trained with hierarchical condition dropout, enabling independent adjustment of content, timbre, and style guidance strengths. Additionally, we employ Representation Alignment (REPA) to distill acoustic-semantic features from a pretrained Whisper model into intermediate DiT representations, stabilizing training and accelerating convergence. Experiments show that DMP-TTS delivers stronger style controllability than open-source baselines while maintaining competitive intelligibility and naturalness. Code and demos will be available at https://y61329697.github.io/DMP-TTS/. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_09504 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | DMP-TTS: Disentangled multi-modal Prompting for Controllable Text-to-Speech with Chained Guidance Yin, Kang Qiang, Chunyu Zhao, Sirui Wang, Xiaopeng Liang, Yuzhe Cai, Pengfei Xu, Tong Zhang, Chen Chen, Enhong Sound Controllable text-to-speech (TTS) systems face significant challenges in achieving independent manipulation of speaker timbre and speaking style, often suffering from entanglement between these attributes. We present DMP-TTS, a latent Diffusion Transformer (DiT) framework with explicit disentanglement and multi-modal prompting. A CLAP-based style encoder (Style-CLAP) aligns cues from reference audio and descriptive text in a shared space and is trained with contrastive learning plus multi-task supervision on style attributes. For fine-grained control during inference, we introduce chained classifier-free guidance (cCFG) trained with hierarchical condition dropout, enabling independent adjustment of content, timbre, and style guidance strengths. Additionally, we employ Representation Alignment (REPA) to distill acoustic-semantic features from a pretrained Whisper model into intermediate DiT representations, stabilizing training and accelerating convergence. Experiments show that DMP-TTS delivers stronger style controllability than open-source baselines while maintaining competitive intelligibility and naturalness. Code and demos will be available at https://y61329697.github.io/DMP-TTS/. |
| title | DMP-TTS: Disentangled multi-modal Prompting for Controllable Text-to-Speech with Chained Guidance |
| topic | Sound |
| url | https://arxiv.org/abs/2512.09504 |