Saved in:
| Main Authors: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2406.08911 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909223164575744 |
|---|---|
| author | Gong, Cheng Cooper, Erica Wang, Xin Qiang, Chunyu Geng, Mengzhe Wells, Dan Wang, Longbiao Dang, Jianwu Tessier, Marc Pine, Aidan Richmond, Korin Yamagishi, Junichi |
| author_facet | Gong, Cheng Cooper, Erica Wang, Xin Qiang, Chunyu Geng, Mengzhe Wells, Dan Wang, Longbiao Dang, Jianwu Tessier, Marc Pine, Aidan Richmond, Korin Yamagishi, Junichi |
| contents | Self-supervised learning (SSL) representations from massively multilingual models offer a promising solution for low-resource language speech tasks. Despite advancements, language adaptation in TTS systems remains an open problem. This paper explores the language adaptation capability of ZMM-TTS, a recent SSL-based multilingual TTS system proposed in our previous work. We conducted experiments on 12 languages using limited data with various fine-tuning configurations. We demonstrate that the similarity in phonetics between the pre-training and target languages, as well as the language category, affects the target language's adaptation performance. Additionally, we find that the fine-tuning dataset size and number of speakers influence adaptability. Surprisingly, we also observed that using paired data for fine-tuning is not always optimal compared to audio-only data. Beyond speech intelligibility, our analysis covers speaker similarity, language identification, and predicted MOS. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_08911 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | An Initial Investigation of Language Adaptation for TTS Systems under Low-resource Scenarios Gong, Cheng Cooper, Erica Wang, Xin Qiang, Chunyu Geng, Mengzhe Wells, Dan Wang, Longbiao Dang, Jianwu Tessier, Marc Pine, Aidan Richmond, Korin Yamagishi, Junichi Computation and Language Audio and Speech Processing Self-supervised learning (SSL) representations from massively multilingual models offer a promising solution for low-resource language speech tasks. Despite advancements, language adaptation in TTS systems remains an open problem. This paper explores the language adaptation capability of ZMM-TTS, a recent SSL-based multilingual TTS system proposed in our previous work. We conducted experiments on 12 languages using limited data with various fine-tuning configurations. We demonstrate that the similarity in phonetics between the pre-training and target languages, as well as the language category, affects the target language's adaptation performance. Additionally, we find that the fine-tuning dataset size and number of speakers influence adaptability. Surprisingly, we also observed that using paired data for fine-tuning is not always optimal compared to audio-only data. Beyond speech intelligibility, our analysis covers speaker similarity, language identification, and predicted MOS. |
| title | An Initial Investigation of Language Adaptation for TTS Systems under Low-resource Scenarios |
| topic | Computation and Language Audio and Speech Processing |
| url | https://arxiv.org/abs/2406.08911 |