DINO-VITS: Data-Efficient Zero-Shot TTS with Self-Supervised Speaker Verification Loss for Noise Robustness
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916292488855552 |
|---|---|
| author | Pankov, Vikentii Pronina, Valeria Kuzmin, Alexander Borisov, Maksim Usoltsev, Nikita Zeng, Xingshan Golubkov, Alexander Ermolenko, Nikolai Shirshova, Aleksandra Matveeva, Yulia |
| author_facet | Pankov, Vikentii Pronina, Valeria Kuzmin, Alexander Borisov, Maksim Usoltsev, Nikita Zeng, Xingshan Golubkov, Alexander Ermolenko, Nikolai Shirshova, Aleksandra Matveeva, Yulia |
| contents | We address zero-shot TTS systems' noise-robustness problem by proposing a dual-objective training for the speaker encoder using self-supervised DINO loss. This approach enhances the speaker encoder with the speech synthesis objective, capturing a wider range of speech characteristics beneficial for voice cloning. At the same time, the DINO objective improves speaker representation learning, ensuring robustness to noise and speaker discriminability. Experiments demonstrate significant improvements in subjective metrics under both clean and noisy conditions, outperforming traditional speaker-encoderbased TTS systems. Additionally, we explore training zeroshot TTS on noisy, unlabeled data. Our two-stage training strategy, leveraging self-supervised speech models to distinguish between noisy and clean speech, shows notable advances in similarity and naturalness, especially with noisy training datasets, compared to the ASR-transcription-based approach. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2311_09770 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | DINO-VITS: Data-Efficient Zero-Shot TTS with Self-Supervised Speaker Verification Loss for Noise Robustness Pankov, Vikentii Pronina, Valeria Kuzmin, Alexander Borisov, Maksim Usoltsev, Nikita Zeng, Xingshan Golubkov, Alexander Ermolenko, Nikolai Shirshova, Aleksandra Matveeva, Yulia Sound Audio and Speech Processing We address zero-shot TTS systems' noise-robustness problem by proposing a dual-objective training for the speaker encoder using self-supervised DINO loss. This approach enhances the speaker encoder with the speech synthesis objective, capturing a wider range of speech characteristics beneficial for voice cloning. At the same time, the DINO objective improves speaker representation learning, ensuring robustness to noise and speaker discriminability. Experiments demonstrate significant improvements in subjective metrics under both clean and noisy conditions, outperforming traditional speaker-encoderbased TTS systems. Additionally, we explore training zeroshot TTS on noisy, unlabeled data. Our two-stage training strategy, leveraging self-supervised speech models to distinguish between noisy and clean speech, shows notable advances in similarity and naturalness, especially with noisy training datasets, compared to the ASR-transcription-based approach. |
| title | DINO-VITS: Data-Efficient Zero-Shot TTS with Self-Supervised Speaker Verification Loss for Noise Robustness |
| topic | Sound Audio and Speech Processing |
| url | https://arxiv.org/abs/2311.09770 |