Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2409.03283 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910908646686720 |
|---|---|
| author | Guo, Hao-Han Hu, Yao Liu, Kun Shen, Fei-Yu Tang, Xu Wu, Yi-Chen Xie, Feng-Long Xie, Kun Xu, Kai-Tuo |
| author_facet | Guo, Hao-Han Hu, Yao Liu, Kun Shen, Fei-Yu Tang, Xu Wu, Yi-Chen Xie, Feng-Long Xie, Kun Xu, Kai-Tuo |
| contents | This work proposes FireRedTTS, a foundation text-to-speech framework, to meet the growing demands for personalized and diverse generative speech applications. The framework comprises three parts: data processing, foundation system, and downstream applications. First, we comprehensively present our data processing pipeline, which transforms massive raw audio into a large-scale high-quality TTS dataset with rich annotations and a wide coverage of content, speaking style, and timbre. Then, we propose a language-model-based foundation TTS system. The speech signal is compressed into discrete semantic tokens via a semantic-aware speech tokenizer, and can be generated by a language model from the prompt text and audio. Then, a two-stage waveform generator is proposed to decode them to the high-fidelity waveform. We present two applications of this system: voice cloning for dubbing and human-like speech generation for chatbots. The experimental results demonstrate the solid in-context learning capability of FireRedTTS, which can stably synthesize high-quality speech consistent with the prompt text and audio. For dubbing, FireRedTTS can clone target voices in a zero-shot way for the UGC scenario and adapt to studio-level expressive voice characters in the PUGC scenario via few-shot fine-tuning with 1-hour recording. Moreover, FireRedTTS achieves controllable human-like speech generation in a casual style with paralinguistic behaviors and emotions via instruction tuning, to better serve spoken chatbots. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_03283 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | FireRedTTS: A Foundation Text-To-Speech Framework for Industry-Level Generative Speech Applications Guo, Hao-Han Hu, Yao Liu, Kun Shen, Fei-Yu Tang, Xu Wu, Yi-Chen Xie, Feng-Long Xie, Kun Xu, Kai-Tuo Sound Audio and Speech Processing This work proposes FireRedTTS, a foundation text-to-speech framework, to meet the growing demands for personalized and diverse generative speech applications. The framework comprises three parts: data processing, foundation system, and downstream applications. First, we comprehensively present our data processing pipeline, which transforms massive raw audio into a large-scale high-quality TTS dataset with rich annotations and a wide coverage of content, speaking style, and timbre. Then, we propose a language-model-based foundation TTS system. The speech signal is compressed into discrete semantic tokens via a semantic-aware speech tokenizer, and can be generated by a language model from the prompt text and audio. Then, a two-stage waveform generator is proposed to decode them to the high-fidelity waveform. We present two applications of this system: voice cloning for dubbing and human-like speech generation for chatbots. The experimental results demonstrate the solid in-context learning capability of FireRedTTS, which can stably synthesize high-quality speech consistent with the prompt text and audio. For dubbing, FireRedTTS can clone target voices in a zero-shot way for the UGC scenario and adapt to studio-level expressive voice characters in the PUGC scenario via few-shot fine-tuning with 1-hour recording. Moreover, FireRedTTS achieves controllable human-like speech generation in a casual style with paralinguistic behaviors and emotions via instruction tuning, to better serve spoken chatbots. |
| title | FireRedTTS: A Foundation Text-To-Speech Framework for Industry-Level Generative Speech Applications |
| topic | Sound Audio and Speech Processing |
| url | https://arxiv.org/abs/2409.03283 |