Accelerating Diffusion-based Text-to-Speech Model Training with Dual Modality Alignment
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866918039806541824 |
|---|---|
| author | Choi, Jeongsoo Niu, Zhikang Kim, Ji-Hoon Wang, Chunhui Chung, Joon Son Chen, Xie |
| author_facet | Choi, Jeongsoo Niu, Zhikang Kim, Ji-Hoon Wang, Chunhui Chung, Joon Son Chen, Xie |
| contents | The goal of this paper is to optimize the training process of diffusion-based text-to-speech models. While recent studies have achieved remarkable advancements, their training demands substantial time and computational costs, largely due to the implicit guidance of diffusion models in learning complex intermediate representations. To address this, we propose A-DMA, an effective strategy for Accelerating training with Dual Modality Alignment. Our method introduces a novel alignment pipeline leveraging both text and speech modalities: text-guided alignment, which incorporates contextual representations, and speech-guided alignment, which refines semantic representations. By aligning hidden states with discriminative features, our training scheme reduces the reliance on diffusion models for learning complex representations. Extensive experiments demonstrate that A-DMA doubles the convergence speed while achieving superior performance over baselines. Code and demo samples are available at: https://github.com/ZhikangNiu/A-DMA |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_19595 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Accelerating Diffusion-based Text-to-Speech Model Training with Dual Modality Alignment Choi, Jeongsoo Niu, Zhikang Kim, Ji-Hoon Wang, Chunhui Chung, Joon Son Chen, Xie Audio and Speech Processing Sound The goal of this paper is to optimize the training process of diffusion-based text-to-speech models. While recent studies have achieved remarkable advancements, their training demands substantial time and computational costs, largely due to the implicit guidance of diffusion models in learning complex intermediate representations. To address this, we propose A-DMA, an effective strategy for Accelerating training with Dual Modality Alignment. Our method introduces a novel alignment pipeline leveraging both text and speech modalities: text-guided alignment, which incorporates contextual representations, and speech-guided alignment, which refines semantic representations. By aligning hidden states with discriminative features, our training scheme reduces the reliance on diffusion models for learning complex representations. Extensive experiments demonstrate that A-DMA doubles the convergence speed while achieving superior performance over baselines. Code and demo samples are available at: https://github.com/ZhikangNiu/A-DMA |
| title | Accelerating Diffusion-based Text-to-Speech Model Training with Dual Modality Alignment |
| topic | Audio and Speech Processing Sound |
| url | https://arxiv.org/abs/2505.19595 |