Generative Speech Foundation Model Pretraining for High-Quality Speech Extraction and Restoration
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909325163757568 |
|---|---|
| author | Ku, Pin-Jui Liu, Alexander H. Korostik, Roman Huang, Sung-Feng Fu, Szu-Wei Jukić, Ante |
| author_facet | Ku, Pin-Jui Liu, Alexander H. Korostik, Roman Huang, Sung-Feng Fu, Szu-Wei Jukić, Ante |
| contents | This paper proposes a generative pretraining foundation model for high-quality speech restoration tasks. By directly operating on complex-valued short-time Fourier transform coefficients, our model does not rely on any vocoders for time-domain signal reconstruction. As a result, our model simplifies the synthesis process and removes the quality upper-bound introduced by any mel-spectrogram vocoder compared to prior work SpeechFlow. The proposed method is evaluated on multiple speech restoration tasks, including speech denoising, bandwidth extension, codec artifact removal, and target speaker extraction. In all scenarios, finetuning our pretrained model results in superior performance over strong baselines. Notably, in the target speaker extraction task, our model outperforms existing systems, including those leveraging SSL-pretrained encoders like WavLM. The code and the pretrained checkpoints are publicly available in the NVIDIA NeMo framework. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_16117 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Generative Speech Foundation Model Pretraining for High-Quality Speech Extraction and Restoration Ku, Pin-Jui Liu, Alexander H. Korostik, Roman Huang, Sung-Feng Fu, Szu-Wei Jukić, Ante Audio and Speech Processing Sound This paper proposes a generative pretraining foundation model for high-quality speech restoration tasks. By directly operating on complex-valued short-time Fourier transform coefficients, our model does not rely on any vocoders for time-domain signal reconstruction. As a result, our model simplifies the synthesis process and removes the quality upper-bound introduced by any mel-spectrogram vocoder compared to prior work SpeechFlow. The proposed method is evaluated on multiple speech restoration tasks, including speech denoising, bandwidth extension, codec artifact removal, and target speaker extraction. In all scenarios, finetuning our pretrained model results in superior performance over strong baselines. Notably, in the target speaker extraction task, our model outperforms existing systems, including those leveraging SSL-pretrained encoders like WavLM. The code and the pretrained checkpoints are publicly available in the NVIDIA NeMo framework. |
| title | Generative Speech Foundation Model Pretraining for High-Quality Speech Extraction and Restoration |
| topic | Audio and Speech Processing Sound |
| url | https://arxiv.org/abs/2409.16117 |