Electrolaryngeal Speech Intelligibility Enhancement Through Robust Linguistic Encoders
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913201626546176 |
|---|---|
| author | Violeta, Lester Phillip Huang, Wen-Chin Ma, Ding Yamamoto, Ryuichi Kobayashi, Kazuhiro Toda, Tomoki |
| author_facet | Violeta, Lester Phillip Huang, Wen-Chin Ma, Ding Yamamoto, Ryuichi Kobayashi, Kazuhiro Toda, Tomoki |
| contents | We propose a novel framework for electrolaryngeal speech intelligibility enhancement through the use of robust linguistic encoders. Pretraining and fine-tuning approaches have proven to work well in this task, but in most cases, various mismatches, such as the speech type mismatch (electrolaryngeal vs. typical) or a speaker mismatch between the datasets used in each stage, can deteriorate the conversion performance of this framework. To resolve this issue, we propose a linguistic encoder robust enough to project both EL and typical speech in the same latent space, while still being able to extract accurate linguistic information, creating a unified representation to reduce the speech type mismatch. Furthermore, we introduce HuBERT output features to the proposed framework for reducing the speaker mismatch, making it possible to effectively use a large-scale parallel dataset during pretraining. We show that compared to the conventional framework using mel-spectrogram input and output features, using the proposed framework enables the model to synthesize more intelligible and naturally sounding speech, as shown by a significant 16% improvement in character error rate and 0.83 improvement in naturalness score. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2309_09627 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Electrolaryngeal Speech Intelligibility Enhancement Through Robust Linguistic Encoders Violeta, Lester Phillip Huang, Wen-Chin Ma, Ding Yamamoto, Ryuichi Kobayashi, Kazuhiro Toda, Tomoki Sound Audio and Speech Processing We propose a novel framework for electrolaryngeal speech intelligibility enhancement through the use of robust linguistic encoders. Pretraining and fine-tuning approaches have proven to work well in this task, but in most cases, various mismatches, such as the speech type mismatch (electrolaryngeal vs. typical) or a speaker mismatch between the datasets used in each stage, can deteriorate the conversion performance of this framework. To resolve this issue, we propose a linguistic encoder robust enough to project both EL and typical speech in the same latent space, while still being able to extract accurate linguistic information, creating a unified representation to reduce the speech type mismatch. Furthermore, we introduce HuBERT output features to the proposed framework for reducing the speaker mismatch, making it possible to effectively use a large-scale parallel dataset during pretraining. We show that compared to the conventional framework using mel-spectrogram input and output features, using the proposed framework enables the model to synthesize more intelligible and naturally sounding speech, as shown by a significant 16% improvement in character error rate and 0.83 improvement in naturalness score. |
| title | Electrolaryngeal Speech Intelligibility Enhancement Through Robust Linguistic Encoders |
| topic | Sound Audio and Speech Processing |
| url | https://arxiv.org/abs/2309.09627 |