Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2506.09349 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917164782452736 |
|---|---|
| author | Tan, Chao-Hong Chen, Qian Wang, Wen Deng, Chong Zhang, Qinglin Cheng, Luyao Yu, Hai Zhang, Xin Lv, Xiang Zhao, Tianyu Zhang, Chong Ma, Yukun Chen, Yafeng Wang, Hui Liu, Jiaqing Li, Xiangang Ye, Jieping |
| author_facet | Tan, Chao-Hong Chen, Qian Wang, Wen Deng, Chong Zhang, Qinglin Cheng, Luyao Yu, Hai Zhang, Xin Lv, Xiang Zhao, Tianyu Zhang, Chong Ma, Yukun Chen, Yafeng Wang, Hui Liu, Jiaqing Li, Xiangang Ye, Jieping |
| contents | Recent studies on end-to-end (E2E) speech generation with large language models (LLMs) have attracted significant community attention, with multiple works extending text-based LLMs to generate discrete speech tokens. Existing E2E approaches primarily fall into two categories: (1) Methods that generate discrete speech tokens independently without incorporating them into the LLM's autoregressive process, resulting in text generation being unaware of concurrent speech synthesis. (2) Models that generate interleaved or parallel speech-text tokens through joint autoregressive modeling, enabling mutual modality awareness during generation. This paper presents DrVoice, a parallel speech-text voice conversation model based on joint autoregressive modeling, featuring dual-resolution speech representations. Notably, while current methods utilize mainly 12.5Hz input audio representation, our proposed dual-resolution mechanism reduces the input frequency for the LLM to 5Hz, significantly reducing computational cost and alleviating the frequency discrepancy between speech and text tokens and in turn better exploiting LLMs' capabilities. Experimental results demonstrate that DrVoice-7B establishes new state-of-the-art (SOTA) on prominent speech benchmarks including OpenAudioBench, VoiceBench, UltraEval-Audio and Big Bench Audio, making it a leading open-source speech foundation model in ~7B models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_09349 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | DrVoice: Parallel Speech-Text Voice Conversation Model via Dual-Resolution Speech Representations Tan, Chao-Hong Chen, Qian Wang, Wen Deng, Chong Zhang, Qinglin Cheng, Luyao Yu, Hai Zhang, Xin Lv, Xiang Zhao, Tianyu Zhang, Chong Ma, Yukun Chen, Yafeng Wang, Hui Liu, Jiaqing Li, Xiangang Ye, Jieping Computation and Language Recent studies on end-to-end (E2E) speech generation with large language models (LLMs) have attracted significant community attention, with multiple works extending text-based LLMs to generate discrete speech tokens. Existing E2E approaches primarily fall into two categories: (1) Methods that generate discrete speech tokens independently without incorporating them into the LLM's autoregressive process, resulting in text generation being unaware of concurrent speech synthesis. (2) Models that generate interleaved or parallel speech-text tokens through joint autoregressive modeling, enabling mutual modality awareness during generation. This paper presents DrVoice, a parallel speech-text voice conversation model based on joint autoregressive modeling, featuring dual-resolution speech representations. Notably, while current methods utilize mainly 12.5Hz input audio representation, our proposed dual-resolution mechanism reduces the input frequency for the LLM to 5Hz, significantly reducing computational cost and alleviating the frequency discrepancy between speech and text tokens and in turn better exploiting LLMs' capabilities. Experimental results demonstrate that DrVoice-7B establishes new state-of-the-art (SOTA) on prominent speech benchmarks including OpenAudioBench, VoiceBench, UltraEval-Audio and Big Bench Audio, making it a leading open-source speech foundation model in ~7B models. |
| title | DrVoice: Parallel Speech-Text Voice Conversation Model via Dual-Resolution Speech Representations |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2506.09349 |