ToneUnit: A Speech Discretization Approach for Tonal Language Speech Synthesis
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866910587990048768 |
|---|---|
| author | Tao, Dehua Tan, Daxin Yeung, Yu Ting Chen, Xiao Lee, Tan |
| author_facet | Tao, Dehua Tan, Daxin Yeung, Yu Ting Chen, Xiao Lee, Tan |
| contents | Representing speech as discretized units has numerous benefits in supporting downstream spoken language processing tasks. However, the approach has been less explored in speech synthesis of tonal languages like Mandarin Chinese. Our preliminary experiments on Chinese speech synthesis reveal the issue of "tone shift", where a synthesized speech utterance contains correct base syllables but incorrect tones. To address the issue, we propose the ToneUnit framework, which leverages annotated data with tone labels as CTC supervision to learn tone-aware discrete speech units for Mandarin Chinese speech. Our findings indicate that the discrete units acquired through the TonUnit resolve the "tone shift" issue in synthesized Chinese speech and yield favorable results in English synthesis. Moreover, the experimental results suggest that finite scalar quantization enhances the effectiveness of ToneUnit. Notably, ToneUnit can work effectively even with minimal annotated data. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_08989 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | ToneUnit: A Speech Discretization Approach for Tonal Language Speech Synthesis Tao, Dehua Tan, Daxin Yeung, Yu Ting Chen, Xiao Lee, Tan Audio and Speech Processing Sound Representing speech as discretized units has numerous benefits in supporting downstream spoken language processing tasks. However, the approach has been less explored in speech synthesis of tonal languages like Mandarin Chinese. Our preliminary experiments on Chinese speech synthesis reveal the issue of "tone shift", where a synthesized speech utterance contains correct base syllables but incorrect tones. To address the issue, we propose the ToneUnit framework, which leverages annotated data with tone labels as CTC supervision to learn tone-aware discrete speech units for Mandarin Chinese speech. Our findings indicate that the discrete units acquired through the TonUnit resolve the "tone shift" issue in synthesized Chinese speech and yield favorable results in English synthesis. Moreover, the experimental results suggest that finite scalar quantization enhances the effectiveness of ToneUnit. Notably, ToneUnit can work effectively even with minimal annotated data. |
| title | ToneUnit: A Speech Discretization Approach for Tonal Language Speech Synthesis |
| topic | Audio and Speech Processing Sound |
| url | https://arxiv.org/abs/2406.08989 |