Thinking-while-speaking: A Controlled, Interleaved Reasoning Method for Real-Time Speech Generation
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2026
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866911701072347136 |
|---|---|
| author | Du, Xuan Yan, Qiangyu Li, Wenshuo Jiang, Borui Xiao, Changming Shu, Han Chen, Xinghao |
| author_facet | Du, Xuan Yan, Qiangyu Li, Wenshuo Jiang, Borui Xiao, Changming Shu, Han Chen, Xinghao |
| contents | The thinking-while-speaking paradigm aims to make AI communication more human. A key challenge is maintaining fluent speech while performing deep reasoning. Our method, InterRS, tackles this by inserting reasoning steps only during natural speech generation. This requires high-quality data where reasoning and speech are precisely aligned, and the length ratio are under controlled. We introduce a novel pipeline to generate such seamlessly interleaved audio data. To train our model, we combine interleaved SFT with refined data and reinforcement learning with two new rewards: a TA-Balance Reward to manage timing and thinking-answer ratio, and a Linguistic Quality Reward to refine expression. Experiments show our approach achieves 13% better performance on mathmatical and logic benchmarks while generating instant response like a spoken-language instruct model which outputs fast CoT response. Furthermore, our method generates more natural and fluent answers than prior methods. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_20946 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Thinking-while-speaking: A Controlled, Interleaved Reasoning Method for Real-Time Speech Generation Du, Xuan Yan, Qiangyu Li, Wenshuo Jiang, Borui Xiao, Changming Shu, Han Chen, Xinghao Computation and Language The thinking-while-speaking paradigm aims to make AI communication more human. A key challenge is maintaining fluent speech while performing deep reasoning. Our method, InterRS, tackles this by inserting reasoning steps only during natural speech generation. This requires high-quality data where reasoning and speech are precisely aligned, and the length ratio are under controlled. We introduce a novel pipeline to generate such seamlessly interleaved audio data. To train our model, we combine interleaved SFT with refined data and reinforcement learning with two new rewards: a TA-Balance Reward to manage timing and thinking-answer ratio, and a Linguistic Quality Reward to refine expression. Experiments show our approach achieves 13% better performance on mathmatical and logic benchmarks while generating instant response like a spoken-language instruct model which outputs fast CoT response. Furthermore, our method generates more natural and fluent answers than prior methods. |
| title | Thinking-while-speaking: A Controlled, Interleaved Reasoning Method for Real-Time Speech Generation |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2605.20946 |