SLIDE: Integrating Speech Language Model with LLM for Spontaneous Spoken Dialogue Generation
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916547277094912 |
|---|---|
| author | Lu, Haitian Cheng, Gaofeng Luo, Liuping Zhang, Leying Qian, Yanmin Zhang, Pengyuan |
| author_facet | Lu, Haitian Cheng, Gaofeng Luo, Liuping Zhang, Leying Qian, Yanmin Zhang, Pengyuan |
| contents | Recently, ``textless" speech language models (SLMs) based on speech units have made huge progress in generating naturalistic speech, including non-verbal vocalizations. However, the generated speech samples often lack semantic coherence. In this paper, we propose SLM and LLM Integration for spontaneous spoken Dialogue gEneration (SLIDE). Specifically, we first utilize an LLM to generate the textual content of spoken dialogue. Next, we convert the textual dialogues into phoneme sequences and use a two-tower transformer-based duration predictor to predict the duration of each phoneme. Finally, an SLM conditioned on the spoken phoneme sequences is used to vocalize the textual dialogue. Experimental results on the Fisher dataset demonstrate that our system can generate naturalistic spoken dialogue while maintaining high semantic coherence. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_00805 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | SLIDE: Integrating Speech Language Model with LLM for Spontaneous Spoken Dialogue Generation Lu, Haitian Cheng, Gaofeng Luo, Liuping Zhang, Leying Qian, Yanmin Zhang, Pengyuan Audio and Speech Processing Computation and Language Sound Recently, ``textless" speech language models (SLMs) based on speech units have made huge progress in generating naturalistic speech, including non-verbal vocalizations. However, the generated speech samples often lack semantic coherence. In this paper, we propose SLM and LLM Integration for spontaneous spoken Dialogue gEneration (SLIDE). Specifically, we first utilize an LLM to generate the textual content of spoken dialogue. Next, we convert the textual dialogues into phoneme sequences and use a two-tower transformer-based duration predictor to predict the duration of each phoneme. Finally, an SLM conditioned on the spoken phoneme sequences is used to vocalize the textual dialogue. Experimental results on the Fisher dataset demonstrate that our system can generate naturalistic spoken dialogue while maintaining high semantic coherence. |
| title | SLIDE: Integrating Speech Language Model with LLM for Spontaneous Spoken Dialogue Generation |
| topic | Audio and Speech Processing Computation and Language Sound |
| url | https://arxiv.org/abs/2501.00805 |