DiffuSpeech: Silent Thought, Spoken Answer via Unified Speech-Text Diffusion
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915763589218304 |
|---|---|
| author | Lou, Yuxuan Wu, Ziming Wang, Yaochen Liu, Yong Ren, Yingxuan Lai, Fuming Lian, Shaobing Tang, Jie You, Yang |
| author_facet | Lou, Yuxuan Wu, Ziming Wang, Yaochen Liu, Yong Ren, Yingxuan Lai, Fuming Lian, Shaobing Tang, Jie You, Yang |
| contents | Current speech language models generate responses directly without explicit reasoning, leading to errors that cannot be corrected once audio is produced. We introduce \textbf{``Silent Thought, Spoken Answer''} -- a paradigm where speech LLMs generate internal text reasoning alongside spoken responses, with thinking traces informing speech quality. To realize this, we present \method{}, the first diffusion-based speech-text language model supporting both understanding and generation, unifying discrete text and tokenized speech under a single masked diffusion framework. Unlike autoregressive approaches, \method{} jointly generates reasoning traces and speech tokens through iterative denoising, with modality-specific masking schedules. We also construct \dataset{}, the first speech QA dataset with paired text reasoning traces, containing 26K samples totaling 319 hours. Experiments show \method{} achieves state-of-the-art speech-to-speech QA accuracy, outperforming the best baseline by up to 9 points, while attaining the best TTS quality among generative models (6.2\% WER) and preserving language understanding (66.2\% MMLU). Ablations confirm that both the diffusion architecture and thinking traces contribute to these gains. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_22889 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | DiffuSpeech: Silent Thought, Spoken Answer via Unified Speech-Text Diffusion Lou, Yuxuan Wu, Ziming Wang, Yaochen Liu, Yong Ren, Yingxuan Lai, Fuming Lian, Shaobing Tang, Jie You, Yang Computation and Language Artificial Intelligence Machine Learning Sound Current speech language models generate responses directly without explicit reasoning, leading to errors that cannot be corrected once audio is produced. We introduce \textbf{``Silent Thought, Spoken Answer''} -- a paradigm where speech LLMs generate internal text reasoning alongside spoken responses, with thinking traces informing speech quality. To realize this, we present \method{}, the first diffusion-based speech-text language model supporting both understanding and generation, unifying discrete text and tokenized speech under a single masked diffusion framework. Unlike autoregressive approaches, \method{} jointly generates reasoning traces and speech tokens through iterative denoising, with modality-specific masking schedules. We also construct \dataset{}, the first speech QA dataset with paired text reasoning traces, containing 26K samples totaling 319 hours. Experiments show \method{} achieves state-of-the-art speech-to-speech QA accuracy, outperforming the best baseline by up to 9 points, while attaining the best TTS quality among generative models (6.2\% WER) and preserving language understanding (66.2\% MMLU). Ablations confirm that both the diffusion architecture and thinking traces contribute to these gains. |
| title | DiffuSpeech: Silent Thought, Spoken Answer via Unified Speech-Text Diffusion |
| topic | Computation and Language Artificial Intelligence Machine Learning Sound |
| url | https://arxiv.org/abs/2601.22889 |