Single-Codec: Single-Codebook Speech Codec towards High-Performance Speech Generation
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909221692375040 |
|---|---|
| author | Li, Hanzhao Xue, Liumeng Guo, Haohan Zhu, Xinfa Lv, Yuanjun Xie, Lei Chen, Yunlin Yin, Hao Li, Zhifei |
| author_facet | Li, Hanzhao Xue, Liumeng Guo, Haohan Zhu, Xinfa Lv, Yuanjun Xie, Lei Chen, Yunlin Yin, Hao Li, Zhifei |
| contents | The multi-codebook speech codec enables the application of large language models (LLM) in TTS but bottlenecks efficiency and robustness due to multi-sequence prediction. To avoid this obstacle, we propose Single-Codec, a single-codebook single-sequence codec, which employs a disentangled VQ-VAE to decouple speech into a time-invariant embedding and a phonetically-rich discrete sequence. Furthermore, the encoder is enhanced with 1) contextual modeling with a BLSTM module to exploit the temporal information, 2) a hybrid sampling module to alleviate distortion from upsampling and downsampling, and 3) a resampling module to encourage discrete units to carry more phonetic information. Compared with multi-codebook codecs, e.g., EnCodec and TiCodec, Single-Codec demonstrates higher reconstruction quality with a lower bandwidth of only 304bps. The effectiveness of Single-Code is further validated by LLM-TTS experiments, showing improved naturalness and intelligibility. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_07422 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Single-Codec: Single-Codebook Speech Codec towards High-Performance Speech Generation Li, Hanzhao Xue, Liumeng Guo, Haohan Zhu, Xinfa Lv, Yuanjun Xie, Lei Chen, Yunlin Yin, Hao Li, Zhifei Audio and Speech Processing The multi-codebook speech codec enables the application of large language models (LLM) in TTS but bottlenecks efficiency and robustness due to multi-sequence prediction. To avoid this obstacle, we propose Single-Codec, a single-codebook single-sequence codec, which employs a disentangled VQ-VAE to decouple speech into a time-invariant embedding and a phonetically-rich discrete sequence. Furthermore, the encoder is enhanced with 1) contextual modeling with a BLSTM module to exploit the temporal information, 2) a hybrid sampling module to alleviate distortion from upsampling and downsampling, and 3) a resampling module to encourage discrete units to carry more phonetic information. Compared with multi-codebook codecs, e.g., EnCodec and TiCodec, Single-Codec demonstrates higher reconstruction quality with a lower bandwidth of only 304bps. The effectiveness of Single-Code is further validated by LLM-TTS experiments, showing improved naturalness and intelligibility. |
| title | Single-Codec: Single-Codebook Speech Codec towards High-Performance Speech Generation |
| topic | Audio and Speech Processing |
| url | https://arxiv.org/abs/2406.07422 |