CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Models
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913626457112576 |
|---|---|
| author | Du, Zhihao Wang, Yuxuan Chen, Qian Shi, Xian Lv, Xiang Zhao, Tianyu Gao, Zhifu Yang, Yexin Gao, Changfeng Wang, Hui Yu, Fan Liu, Huadai Sheng, Zhengyan Gu, Yue Deng, Chong Wang, Wen Zhang, Shiliang Yan, Zhijie Zhou, Jingren |
| author_facet | Du, Zhihao Wang, Yuxuan Chen, Qian Shi, Xian Lv, Xiang Zhao, Tianyu Gao, Zhifu Yang, Yexin Gao, Changfeng Wang, Hui Yu, Fan Liu, Huadai Sheng, Zhengyan Gu, Yue Deng, Chong Wang, Wen Zhang, Shiliang Yan, Zhijie Zhou, Jingren |
| contents | In our previous work, we introduced CosyVoice, a multilingual speech synthesis model based on supervised discrete speech tokens. By employing progressive semantic decoding with two popular generative models, language models (LMs) and Flow Matching, CosyVoice demonstrated high prosody naturalness, content consistency, and speaker similarity in speech in-context learning. Recently, significant progress has been made in multi-modal large language models (LLMs), where the response latency and real-time factor of speech synthesis play a crucial role in the interactive experience. Therefore, in this report, we present an improved streaming speech synthesis model, CosyVoice 2, which incorporates comprehensive and systematic optimizations. Specifically, we introduce finite-scalar quantization to improve the codebook utilization of speech tokens. For the text-speech LM, we streamline the model architecture to allow direct use of a pre-trained LLM as the backbone. In addition, we develop a chunk-aware causal flow matching model to support various synthesis scenarios, enabling both streaming and non-streaming synthesis within a single model. By training on a large-scale multilingual dataset, CosyVoice 2 achieves human-parity naturalness, minimal response latency, and virtually lossless synthesis quality in the streaming mode. We invite readers to listen to the demos at https://funaudiollm.github.io/cosyvoice2. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_10117 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Models Du, Zhihao Wang, Yuxuan Chen, Qian Shi, Xian Lv, Xiang Zhao, Tianyu Gao, Zhifu Yang, Yexin Gao, Changfeng Wang, Hui Yu, Fan Liu, Huadai Sheng, Zhengyan Gu, Yue Deng, Chong Wang, Wen Zhang, Shiliang Yan, Zhijie Zhou, Jingren Sound Artificial Intelligence Machine Learning Audio and Speech Processing In our previous work, we introduced CosyVoice, a multilingual speech synthesis model based on supervised discrete speech tokens. By employing progressive semantic decoding with two popular generative models, language models (LMs) and Flow Matching, CosyVoice demonstrated high prosody naturalness, content consistency, and speaker similarity in speech in-context learning. Recently, significant progress has been made in multi-modal large language models (LLMs), where the response latency and real-time factor of speech synthesis play a crucial role in the interactive experience. Therefore, in this report, we present an improved streaming speech synthesis model, CosyVoice 2, which incorporates comprehensive and systematic optimizations. Specifically, we introduce finite-scalar quantization to improve the codebook utilization of speech tokens. For the text-speech LM, we streamline the model architecture to allow direct use of a pre-trained LLM as the backbone. In addition, we develop a chunk-aware causal flow matching model to support various synthesis scenarios, enabling both streaming and non-streaming synthesis within a single model. By training on a large-scale multilingual dataset, CosyVoice 2 achieves human-parity naturalness, minimal response latency, and virtually lossless synthesis quality in the streaming mode. We invite readers to listen to the demos at https://funaudiollm.github.io/cosyvoice2. |
| title | CosyVoice 2: Scalable Streaming Speech Synthesis with Large Language Models |
| topic | Sound Artificial Intelligence Machine Learning Audio and Speech Processing |
| url | https://arxiv.org/abs/2412.10117 |