Phonetic Enhanced Language Modeling for Text-to-Speech Synthesis
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866910482696241152 |
|---|---|
| author | Zhou, Kun Zhao, Shengkui Ma, Yukun Zhang, Chong Wang, Hao Ng, Dianwen Ni, Chongjia Hieu, Nguyen Trung Yip, Jia Qi Ma, Bin |
| author_facet | Zhou, Kun Zhao, Shengkui Ma, Yukun Zhang, Chong Wang, Hao Ng, Dianwen Ni, Chongjia Hieu, Nguyen Trung Yip, Jia Qi Ma, Bin |
| contents | Recent language model-based text-to-speech (TTS) frameworks demonstrate scalability and in-context learning capabilities. However, they suffer from robustness issues due to the accumulation of errors in speech unit predictions during autoregressive language modeling. In this paper, we propose a phonetic enhanced language modeling method to improve the performance of TTS models. We leverage self-supervised representations that are phonetically rich as the training target for the autoregressive language model. Subsequently, a non-autoregressive model is employed to predict discrete acoustic codecs that contain fine-grained acoustic details. The TTS model focuses solely on linguistic modeling during autoregressive training, thereby reducing the error propagation that occurs in non-autoregressive training. Both objective and subjective evaluations validate the effectiveness of our proposed method. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_02009 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Phonetic Enhanced Language Modeling for Text-to-Speech Synthesis Zhou, Kun Zhao, Shengkui Ma, Yukun Zhang, Chong Wang, Hao Ng, Dianwen Ni, Chongjia Hieu, Nguyen Trung Yip, Jia Qi Ma, Bin Audio and Speech Processing Computation and Language Sound Recent language model-based text-to-speech (TTS) frameworks demonstrate scalability and in-context learning capabilities. However, they suffer from robustness issues due to the accumulation of errors in speech unit predictions during autoregressive language modeling. In this paper, we propose a phonetic enhanced language modeling method to improve the performance of TTS models. We leverage self-supervised representations that are phonetically rich as the training target for the autoregressive language model. Subsequently, a non-autoregressive model is employed to predict discrete acoustic codecs that contain fine-grained acoustic details. The TTS model focuses solely on linguistic modeling during autoregressive training, thereby reducing the error propagation that occurs in non-autoregressive training. Both objective and subjective evaluations validate the effectiveness of our proposed method. |
| title | Phonetic Enhanced Language Modeling for Text-to-Speech Synthesis |
| topic | Audio and Speech Processing Computation and Language Sound |
| url | https://arxiv.org/abs/2406.02009 |