Investigating Neural Audio Codecs for Speech Language Model-Based Speech Generation
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866929489582227456 |
|---|---|
| author | Li, Jiaqi Wang, Dongmei Wang, Xiaofei Qian, Yao Zhou, Long Liu, Shujie Yousefi, Midia Li, Canrun Tsai, Chung-Hsien Xiao, Zhen Liu, Yanqing Chen, Junkun Zhao, Sheng Li, Jinyu Wu, Zhizheng Zeng, Michael |
| author_facet | Li, Jiaqi Wang, Dongmei Wang, Xiaofei Qian, Yao Zhou, Long Liu, Shujie Yousefi, Midia Li, Canrun Tsai, Chung-Hsien Xiao, Zhen Liu, Yanqing Chen, Junkun Zhao, Sheng Li, Jinyu Wu, Zhizheng Zeng, Michael |
| contents | Neural audio codec tokens serve as the fundamental building blocks for speech language model (SLM)-based speech generation. However, there is no systematic understanding on how the codec system affects the speech generation performance of the SLM. In this work, we examine codec tokens within SLM framework for speech generation to provide insights for effective codec design. We retrain existing high-performing neural codec models on the same data set and loss functions to compare their performance in a uniform setting. We integrate codec tokens into two SLM systems: masked-based parallel speech generation system and an auto-regressive (AR) plus non-auto-regressive (NAR) model-based system. Our findings indicate that better speech reconstruction in codec systems does not guarantee improved speech generation in SLM. A high-quality codec decoder is crucial for natural speech production in SLM, while speech intelligibility depends more on quantization mechanism. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_04016 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Investigating Neural Audio Codecs for Speech Language Model-Based Speech Generation Li, Jiaqi Wang, Dongmei Wang, Xiaofei Qian, Yao Zhou, Long Liu, Shujie Yousefi, Midia Li, Canrun Tsai, Chung-Hsien Xiao, Zhen Liu, Yanqing Chen, Junkun Zhao, Sheng Li, Jinyu Wu, Zhizheng Zeng, Michael Sound Audio and Speech Processing Neural audio codec tokens serve as the fundamental building blocks for speech language model (SLM)-based speech generation. However, there is no systematic understanding on how the codec system affects the speech generation performance of the SLM. In this work, we examine codec tokens within SLM framework for speech generation to provide insights for effective codec design. We retrain existing high-performing neural codec models on the same data set and loss functions to compare their performance in a uniform setting. We integrate codec tokens into two SLM systems: masked-based parallel speech generation system and an auto-regressive (AR) plus non-auto-regressive (NAR) model-based system. Our findings indicate that better speech reconstruction in codec systems does not guarantee improved speech generation in SLM. A high-quality codec decoder is crucial for natural speech production in SLM, while speech intelligibility depends more on quantization mechanism. |
| title | Investigating Neural Audio Codecs for Speech Language Model-Based Speech Generation |
| topic | Sound Audio and Speech Processing |
| url | https://arxiv.org/abs/2409.04016 |