An Empirical Study of Speech Language Models for Prompt-Conditioned Speech Synthesis
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866916165881692160 |
|---|---|
| author | Peng, Yifan Kulikov, Ilia Yang, Yilin Popuri, Sravya Lu, Hui Wang, Changhan Gong, Hongyu |
| author_facet | Peng, Yifan Kulikov, Ilia Yang, Yilin Popuri, Sravya Lu, Hui Wang, Changhan Gong, Hongyu |
| contents | Speech language models (LMs) are promising for high-quality speech synthesis through in-context learning. A typical speech LM takes discrete semantic units as content and a short utterance as prompt, and synthesizes speech which preserves the content's semantics but mimics the prompt's style. However, there is no systematic understanding on how the synthesized audio is controlled by the prompt and content. In this work, we conduct an empirical study of the widely used autoregressive (AR) and non-autoregressive (NAR) speech LMs and provide insights into the prompt design and content semantic units. Our analysis reveals that heterogeneous and nonstationary prompts hurt the audio quality in contrast to the previous finding that longer prompts always lead to better synthesis. Moreover, we find that the speaker style of the synthesized audio is also affected by the content in addition to the prompt. We further show that semantic units carry rich acoustic information such as pitch, tempo, volume and speech emphasis, which might be leaked from the content to the synthesized audio. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_12402 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | An Empirical Study of Speech Language Models for Prompt-Conditioned Speech Synthesis Peng, Yifan Kulikov, Ilia Yang, Yilin Popuri, Sravya Lu, Hui Wang, Changhan Gong, Hongyu Computation and Language Sound Audio and Speech Processing Speech language models (LMs) are promising for high-quality speech synthesis through in-context learning. A typical speech LM takes discrete semantic units as content and a short utterance as prompt, and synthesizes speech which preserves the content's semantics but mimics the prompt's style. However, there is no systematic understanding on how the synthesized audio is controlled by the prompt and content. In this work, we conduct an empirical study of the widely used autoregressive (AR) and non-autoregressive (NAR) speech LMs and provide insights into the prompt design and content semantic units. Our analysis reveals that heterogeneous and nonstationary prompts hurt the audio quality in contrast to the previous finding that longer prompts always lead to better synthesis. Moreover, we find that the speaker style of the synthesized audio is also affected by the content in addition to the prompt. We further show that semantic units carry rich acoustic information such as pitch, tempo, volume and speech emphasis, which might be leaked from the content to the synthesized audio. |
| title | An Empirical Study of Speech Language Models for Prompt-Conditioned Speech Synthesis |
| topic | Computation and Language Sound Audio and Speech Processing |
| url | https://arxiv.org/abs/2403.12402 |