InSerter: Speech Instruction Following with Unsupervised Interleaved Pre-training
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866910985312272384 |
|---|---|
| author | Wang, Dingdong Xu, Jin Chu, Ruihang Guo, Zhifang Wang, Xiong Wu, Jincenzi Yang, Dongchao Ji, Shengpeng Lin, Junyang |
| author_facet | Wang, Dingdong Xu, Jin Chu, Ruihang Guo, Zhifang Wang, Xiong Wu, Jincenzi Yang, Dongchao Ji, Shengpeng Lin, Junyang |
| contents | Recent advancements in speech large language models (SpeechLLMs) have attracted considerable attention. Nonetheless, current methods exhibit suboptimal performance in adhering to speech instructions. Notably, the intelligence of models significantly diminishes when processing speech-form input as compared to direct text-form input. Prior work has attempted to mitigate this semantic inconsistency between speech and text representations through techniques such as representation and behavior alignment, which involve the meticulous design of data pairs during the post-training phase. In this paper, we introduce a simple and scalable training method called InSerter, which stands for Interleaved Speech-Text Representation Pre-training. InSerter is designed to pre-train large-scale unsupervised speech-text sequences, where the speech is synthesized from randomly selected segments of an extensive text corpus using text-to-speech conversion. Consequently, the model acquires the ability to generate textual continuations corresponding to the provided speech segments, obviating the need for intensive data design endeavors. To systematically evaluate speech instruction-following capabilities, we introduce SpeechInstructBench, the first comprehensive benchmark specifically designed for speech-oriented instruction-following tasks. Our proposed InSerter achieves SOTA performance in SpeechInstructBench and demonstrates superior or competitive results across diverse speech processing tasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_02769 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | InSerter: Speech Instruction Following with Unsupervised Interleaved Pre-training Wang, Dingdong Xu, Jin Chu, Ruihang Guo, Zhifang Wang, Xiong Wu, Jincenzi Yang, Dongchao Ji, Shengpeng Lin, Junyang Sound Computation and Language Human-Computer Interaction Audio and Speech Processing Recent advancements in speech large language models (SpeechLLMs) have attracted considerable attention. Nonetheless, current methods exhibit suboptimal performance in adhering to speech instructions. Notably, the intelligence of models significantly diminishes when processing speech-form input as compared to direct text-form input. Prior work has attempted to mitigate this semantic inconsistency between speech and text representations through techniques such as representation and behavior alignment, which involve the meticulous design of data pairs during the post-training phase. In this paper, we introduce a simple and scalable training method called InSerter, which stands for Interleaved Speech-Text Representation Pre-training. InSerter is designed to pre-train large-scale unsupervised speech-text sequences, where the speech is synthesized from randomly selected segments of an extensive text corpus using text-to-speech conversion. Consequently, the model acquires the ability to generate textual continuations corresponding to the provided speech segments, obviating the need for intensive data design endeavors. To systematically evaluate speech instruction-following capabilities, we introduce SpeechInstructBench, the first comprehensive benchmark specifically designed for speech-oriented instruction-following tasks. Our proposed InSerter achieves SOTA performance in SpeechInstructBench and demonstrates superior or competitive results across diverse speech processing tasks. |
| title | InSerter: Speech Instruction Following with Unsupervised Interleaved Pre-training |
| topic | Sound Computation and Language Human-Computer Interaction Audio and Speech Processing |
| url | https://arxiv.org/abs/2503.02769 |