STITCH: Simultaneous Thinking and Talking with Chunked Reasoning for Spoken Language Models
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866917258350034944 |
|---|---|
| author | Chiang, Cheng-Han Wang, Xiaofei Li, Linjie Lin, Chung-Ching Lin, Kevin Liu, Shujie Wang, Zhendong Yang, Zhengyuan Lee, Hung-yi Wang, Lijuan |
| author_facet | Chiang, Cheng-Han Wang, Xiaofei Li, Linjie Lin, Chung-Ching Lin, Kevin Liu, Shujie Wang, Zhendong Yang, Zhengyuan Lee, Hung-yi Wang, Lijuan |
| contents | Spoken Language Models (SLMs) are designed to take speech inputs and produce spoken responses. However, current SLMs lack the ability to perform an internal, unspoken thinking process before responding. In contrast, humans typically engage in complex mental reasoning internally, enabling them to communicate ideas clearly and concisely. Thus, integrating an unspoken thought process into SLMs is highly desirable. While naively generating a complete chain-of-thought (CoT) reasoning before starting to talk can enable thinking for SLMs, this induces additional latency for the speech response, as the CoT reasoning can be arbitrarily long. To solve this issue, we propose Stitch, a novel generation method that alternates between the generation of unspoken reasoning chunks and spoken response chunks. Since the audio duration of a chunk of spoken response is much longer than the time to generate the tokens in a chunk of spoken response, we use the remaining free time to generate the unspoken reasoning tokens. When a chunk of audio is played to the user, the model continues to generate the next unspoken reasoning chunk, achieving simultaneous thinking and talking. Remarkably, Stitch matches the latency of baselines that cannot generate unspoken CoT by design while outperforming those baselines by 15% on math reasoning datasets; Stitch also performs equally well on non-reasoning datasets as those baseline models. Some animations and demonstrations are on the project page: https://d223302.github.io/STITCH. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_15375 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | STITCH: Simultaneous Thinking and Talking with Chunked Reasoning for Spoken Language Models Chiang, Cheng-Han Wang, Xiaofei Li, Linjie Lin, Chung-Ching Lin, Kevin Liu, Shujie Wang, Zhendong Yang, Zhengyuan Lee, Hung-yi Wang, Lijuan Computation and Language Audio and Speech Processing Spoken Language Models (SLMs) are designed to take speech inputs and produce spoken responses. However, current SLMs lack the ability to perform an internal, unspoken thinking process before responding. In contrast, humans typically engage in complex mental reasoning internally, enabling them to communicate ideas clearly and concisely. Thus, integrating an unspoken thought process into SLMs is highly desirable. While naively generating a complete chain-of-thought (CoT) reasoning before starting to talk can enable thinking for SLMs, this induces additional latency for the speech response, as the CoT reasoning can be arbitrarily long. To solve this issue, we propose Stitch, a novel generation method that alternates between the generation of unspoken reasoning chunks and spoken response chunks. Since the audio duration of a chunk of spoken response is much longer than the time to generate the tokens in a chunk of spoken response, we use the remaining free time to generate the unspoken reasoning tokens. When a chunk of audio is played to the user, the model continues to generate the next unspoken reasoning chunk, achieving simultaneous thinking and talking. Remarkably, Stitch matches the latency of baselines that cannot generate unspoken CoT by design while outperforming those baselines by 15% on math reasoning datasets; Stitch also performs equally well on non-reasoning datasets as those baseline models. Some animations and demonstrations are on the project page: https://d223302.github.io/STITCH. |
| title | STITCH: Simultaneous Thinking and Talking with Chunked Reasoning for Spoken Language Models |
| topic | Computation and Language Audio and Speech Processing |
| url | https://arxiv.org/abs/2507.15375 |