FreeCodec: A disentangled neural speech codec with fewer tokens
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866911026882019328 |
|---|---|
| author | Zheng, Youqiang Tu, Weiping Kang, Yueteng Chen, Jie Zhang, Yike Xiao, Li Yang, Yuhong Ma, Long |
| author_facet | Zheng, Youqiang Tu, Weiping Kang, Yueteng Chen, Jie Zhang, Yike Xiao, Li Yang, Yuhong Ma, Long |
| contents | Neural speech codecs have gained great attention for their outstanding reconstruction with discrete token representations.
It is a crucial component in generative tasks such as speech coding and large language models (LLM).
However, most works based on residual vector quantization perform worse with fewer tokens due to low coding efficiency for modeling complex coupled information.
In this paper, we propose a neural speech codec named FreeCodec which employs a more effective encoding framework by decomposing intrinsic properties of speech into different components:
1) a global vector is extracted as the timbre information,
2) a prosody encoder with a long stride level is used to model the prosody information,
3) the content information is from a content encoder.
Using different training strategies, FreeCodec achieves state-of-the-art performance in reconstruction and disentanglement scenarios.
Results from subjective and objective experiments demonstrate that our framework outperforms existing methods. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_01053 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | FreeCodec: A disentangled neural speech codec with fewer tokens Zheng, Youqiang Tu, Weiping Kang, Yueteng Chen, Jie Zhang, Yike Xiao, Li Yang, Yuhong Ma, Long Sound Audio and Speech Processing Neural speech codecs have gained great attention for their outstanding reconstruction with discrete token representations. It is a crucial component in generative tasks such as speech coding and large language models (LLM). However, most works based on residual vector quantization perform worse with fewer tokens due to low coding efficiency for modeling complex coupled information. In this paper, we propose a neural speech codec named FreeCodec which employs a more effective encoding framework by decomposing intrinsic properties of speech into different components: 1) a global vector is extracted as the timbre information, 2) a prosody encoder with a long stride level is used to model the prosody information, 3) the content information is from a content encoder. Using different training strategies, FreeCodec achieves state-of-the-art performance in reconstruction and disentanglement scenarios. Results from subjective and objective experiments demonstrate that our framework outperforms existing methods. |
| title | FreeCodec: A disentangled neural speech codec with fewer tokens |
| topic | Sound Audio and Speech Processing |
| url | https://arxiv.org/abs/2412.01053 |