Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2409.12139 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909323771248640 |
|---|---|
| author | Chen, Sijing Feng, Yuan He, Laipeng He, Tianwei He, Wendi Hu, Yanni Lin, Bin Lin, Yiting Pan, Yu Tan, Pengfei Tian, Chengwei Wang, Chen Wang, Zhicheng Xie, Ruoye Yao, Jixun Yan, Quanlei Yang, Yuguang Ye, Jianhao Yin, Jingjing Yu, Yanzhen Zhang, Huimin Zhang, Xiang Zhao, Guangcheng Zhou, Hongbin Zou, Pengpeng |
| author_facet | Chen, Sijing Feng, Yuan He, Laipeng He, Tianwei He, Wendi Hu, Yanni Lin, Bin Lin, Yiting Pan, Yu Tan, Pengfei Tian, Chengwei Wang, Chen Wang, Zhicheng Xie, Ruoye Yao, Jixun Yan, Quanlei Yang, Yuguang Ye, Jianhao Yin, Jingjing Yu, Yanzhen Zhang, Huimin Zhang, Xiang Zhao, Guangcheng Zhou, Hongbin Zou, Pengpeng |
| contents | With the advent of the big data and large language model era, zero-shot personalized rapid customization has emerged as a significant trend. In this report, we introduce Takin AudioLLM, a series of techniques and models, mainly including Takin TTS, Takin VC, and Takin Morphing, specifically designed for audiobook production. These models are capable of zero-shot speech production, generating high-quality speech that is nearly indistinguishable from real human speech and facilitating individuals to customize the speech content according to their own needs. Specifically, we first introduce Takin TTS, a neural codec language model that builds upon an enhanced neural speech codec and a multi-task training framework, capable of generating high-fidelity natural speech in a zero-shot way. For Takin VC, we advocate an effective content and timbre joint modeling approach to improve the speaker similarity, while advocating for a conditional flow matching based decoder to further enhance its naturalness and expressiveness. Last, we propose the Takin Morphing system with highly decoupled and advanced timbre and prosody modeling approaches, which enables individuals to customize speech production with their preferred timbre and prosody in a precise and controllable manner. Extensive experiments validate the effectiveness and robustness of our Takin AudioLLM series models. For detailed demos, please refer to https://everest-ai.github.io/takinaudiollm/. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_12139 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Takin: A Cohort of Superior Quality Zero-shot Speech Generation Models Chen, Sijing Feng, Yuan He, Laipeng He, Tianwei He, Wendi Hu, Yanni Lin, Bin Lin, Yiting Pan, Yu Tan, Pengfei Tian, Chengwei Wang, Chen Wang, Zhicheng Xie, Ruoye Yao, Jixun Yan, Quanlei Yang, Yuguang Ye, Jianhao Yin, Jingjing Yu, Yanzhen Zhang, Huimin Zhang, Xiang Zhao, Guangcheng Zhou, Hongbin Zou, Pengpeng Sound Artificial Intelligence Audio and Speech Processing With the advent of the big data and large language model era, zero-shot personalized rapid customization has emerged as a significant trend. In this report, we introduce Takin AudioLLM, a series of techniques and models, mainly including Takin TTS, Takin VC, and Takin Morphing, specifically designed for audiobook production. These models are capable of zero-shot speech production, generating high-quality speech that is nearly indistinguishable from real human speech and facilitating individuals to customize the speech content according to their own needs. Specifically, we first introduce Takin TTS, a neural codec language model that builds upon an enhanced neural speech codec and a multi-task training framework, capable of generating high-fidelity natural speech in a zero-shot way. For Takin VC, we advocate an effective content and timbre joint modeling approach to improve the speaker similarity, while advocating for a conditional flow matching based decoder to further enhance its naturalness and expressiveness. Last, we propose the Takin Morphing system with highly decoupled and advanced timbre and prosody modeling approaches, which enables individuals to customize speech production with their preferred timbre and prosody in a precise and controllable manner. Extensive experiments validate the effectiveness and robustness of our Takin AudioLLM series models. For detailed demos, please refer to https://everest-ai.github.io/takinaudiollm/. |
| title | Takin: A Cohort of Superior Quality Zero-shot Speech Generation Models |
| topic | Sound Artificial Intelligence Audio and Speech Processing |
| url | https://arxiv.org/abs/2409.12139 |