IntrinsicVoice: Empowering LLMs with Intrinsic Real-time Voice Interaction Abilities
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866909346643836928 |
|---|---|
| author | Zhang, Xin Lyu, Xiang Du, Zhihao Chen, Qian Zhang, Dong Hu, Hangrui Tan, Chaohong Zhao, Tianyu Wang, Yuxuan Zhang, Bin Lu, Heng Zhou, Yaqian Qiu, Xipeng |
| author_facet | Zhang, Xin Lyu, Xiang Du, Zhihao Chen, Qian Zhang, Dong Hu, Hangrui Tan, Chaohong Zhao, Tianyu Wang, Yuxuan Zhang, Bin Lu, Heng Zhou, Yaqian Qiu, Xipeng |
| contents | Current methods of building LLMs with voice interaction capabilities rely heavily on explicit text autoregressive generation before or during speech response generation to maintain content quality, which unfortunately brings computational overhead and increases latency in multi-turn interactions. To address this, we introduce IntrinsicVoic,e an LLM designed with intrinsic real-time voice interaction capabilities. IntrinsicVoice aims to facilitate the transfer of textual capabilities of pre-trained LLMs to the speech modality by mitigating the modality gap between text and speech. Our novelty architecture, GroupFormer, can reduce speech sequences to lengths comparable to text sequences while generating high-quality audio, significantly reducing the length difference between speech and text, speeding up inference, and alleviating long-text modeling issues. Additionally, we construct a multi-turn speech-to-speech dialogue dataset named \method-500k which includes nearly 500k turns of speech-to-speech dialogues, and a cross-modality training strategy to enhance the semantic alignment between speech and text. Experimental results demonstrate that IntrinsicVoice can generate high-quality speech response with latency lower than 100ms in multi-turn dialogue scenarios. Demos are available at https://instrinsicvoice.github.io/. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_08035 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | IntrinsicVoice: Empowering LLMs with Intrinsic Real-time Voice Interaction Abilities Zhang, Xin Lyu, Xiang Du, Zhihao Chen, Qian Zhang, Dong Hu, Hangrui Tan, Chaohong Zhao, Tianyu Wang, Yuxuan Zhang, Bin Lu, Heng Zhou, Yaqian Qiu, Xipeng Sound Artificial Intelligence Current methods of building LLMs with voice interaction capabilities rely heavily on explicit text autoregressive generation before or during speech response generation to maintain content quality, which unfortunately brings computational overhead and increases latency in multi-turn interactions. To address this, we introduce IntrinsicVoic,e an LLM designed with intrinsic real-time voice interaction capabilities. IntrinsicVoice aims to facilitate the transfer of textual capabilities of pre-trained LLMs to the speech modality by mitigating the modality gap between text and speech. Our novelty architecture, GroupFormer, can reduce speech sequences to lengths comparable to text sequences while generating high-quality audio, significantly reducing the length difference between speech and text, speeding up inference, and alleviating long-text modeling issues. Additionally, we construct a multi-turn speech-to-speech dialogue dataset named \method-500k which includes nearly 500k turns of speech-to-speech dialogues, and a cross-modality training strategy to enhance the semantic alignment between speech and text. Experimental results demonstrate that IntrinsicVoice can generate high-quality speech response with latency lower than 100ms in multi-turn dialogue scenarios. Demos are available at https://instrinsicvoice.github.io/. |
| title | IntrinsicVoice: Empowering LLMs with Intrinsic Real-time Voice Interaction Abilities |
| topic | Sound Artificial Intelligence |
| url | https://arxiv.org/abs/2410.08035 |