Discrete Diffusion for Generative Modeling of Text-Aligned Speech Tokens
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866916967638630400 |
|---|---|
| author | Ku, Pin-Jui Huang, He Lemercier, Jean-Marie Sahoo, Subham Sekhar Chen, Zhehuai Jukić, Ante |
| author_facet | Ku, Pin-Jui Huang, He Lemercier, Jean-Marie Sahoo, Subham Sekhar Chen, Zhehuai Jukić, Ante |
| contents | This paper introduces a discrete diffusion model (DDM) framework for text-aligned speech tokenization and reconstruction. By replacing the auto-regressive speech decoder with a discrete diffusion counterpart, our model achieves significantly better reconstruction quality, stronger ASR performance, and faster inference. We provide a comprehensive analysis of applying DDMs to speech reconstruction, examining sampler choices, inference steps, and robustness to length-scale estimation errors. Furthermore, we improve the original TASTE by systematically comparing vector quantization modules, showing that FSQ yields up to a 35% relative WER reduction and +0.14 UT-MOS improvement over RVQ for AR models, while also enhancing DDM performance. Our model generates speech in just 10 denoising steps and even supports single-step generation with only minor quality degradation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_20060 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Discrete Diffusion for Generative Modeling of Text-Aligned Speech Tokens Ku, Pin-Jui Huang, He Lemercier, Jean-Marie Sahoo, Subham Sekhar Chen, Zhehuai Jukić, Ante Audio and Speech Processing This paper introduces a discrete diffusion model (DDM) framework for text-aligned speech tokenization and reconstruction. By replacing the auto-regressive speech decoder with a discrete diffusion counterpart, our model achieves significantly better reconstruction quality, stronger ASR performance, and faster inference. We provide a comprehensive analysis of applying DDMs to speech reconstruction, examining sampler choices, inference steps, and robustness to length-scale estimation errors. Furthermore, we improve the original TASTE by systematically comparing vector quantization modules, showing that FSQ yields up to a 35% relative WER reduction and +0.14 UT-MOS improvement over RVQ for AR models, while also enhancing DDM performance. Our model generates speech in just 10 denoising steps and even supports single-step generation with only minor quality degradation. |
| title | Discrete Diffusion for Generative Modeling of Text-Aligned Speech Tokens |
| topic | Audio and Speech Processing |
| url | https://arxiv.org/abs/2509.20060 |