Towards Efficient Speech-Text Jointly Decoding within One Speech Language Model
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866912895420334080 |
|---|---|
| author | Wu, Haibin Hu, Yuxuan Fan, Ruchao Wang, Xiaofei Kumatani, Kenichi Ren, Bo Yu, Jianwei Lu, Heng Wang, Lijuan Qian, Yao Li, Jinyu |
| author_facet | Wu, Haibin Hu, Yuxuan Fan, Ruchao Wang, Xiaofei Kumatani, Kenichi Ren, Bo Yu, Jianwei Lu, Heng Wang, Lijuan Qian, Yao Li, Jinyu |
| contents | Speech language models (Speech LMs) enable end-to-end speech-text modeling within a single model, offering a promising direction for spoken dialogue systems. The choice of speech-text jointly decoding paradigm plays a critical role in performance, efficiency, and alignment quality. In this work, we systematically compare representative joint speech-text decoding strategies, including the interleaved, and parallel generation paradigms, under a controlled experimental setup using the same base language model, speech tokenizer and training data. Our results show that the interleaved approach achieves the best alignment. However it suffers from slow inference due to long token sequence length. To address this, we propose a novel early-stop interleaved (ESI) pattern that not only significantly accelerates decoding but also yields slightly better performance. Additionally, we curate high-quality question answering (QA) datasets to further improve speech QA performance. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_04518 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Towards Efficient Speech-Text Jointly Decoding within One Speech Language Model Wu, Haibin Hu, Yuxuan Fan, Ruchao Wang, Xiaofei Kumatani, Kenichi Ren, Bo Yu, Jianwei Lu, Heng Wang, Lijuan Qian, Yao Li, Jinyu Audio and Speech Processing Computation and Language Speech language models (Speech LMs) enable end-to-end speech-text modeling within a single model, offering a promising direction for spoken dialogue systems. The choice of speech-text jointly decoding paradigm plays a critical role in performance, efficiency, and alignment quality. In this work, we systematically compare representative joint speech-text decoding strategies, including the interleaved, and parallel generation paradigms, under a controlled experimental setup using the same base language model, speech tokenizer and training data. Our results show that the interleaved approach achieves the best alignment. However it suffers from slow inference due to long token sequence length. To address this, we propose a novel early-stop interleaved (ESI) pattern that not only significantly accelerates decoding but also yields slightly better performance. Additionally, we curate high-quality question answering (QA) datasets to further improve speech QA performance. |
| title | Towards Efficient Speech-Text Jointly Decoding within One Speech Language Model |
| topic | Audio and Speech Processing Computation and Language |
| url | https://arxiv.org/abs/2506.04518 |