Noise-Robust Target-Speaker Voice Activity Detection Through Self-Supervised Pretraining
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909448997437440 |
|---|---|
| author | Bovbjerg, Holger Severin Østergaard, Jan Jensen, Jesper Tan, Zheng-Hua |
| author_facet | Bovbjerg, Holger Severin Østergaard, Jan Jensen, Jesper Tan, Zheng-Hua |
| contents | Target-Speaker Voice Activity Detection (TS-VAD) is the task of detecting the presence of speech from a known target-speaker in an audio frame. Recently, deep neural network-based models have shown good performance in this task. However, training these models requires extensive labelled data, which is costly and time-consuming to obtain, particularly if generalization to unseen environments is crucial. To mitigate this, we propose a causal, Self-Supervised Learning (SSL) pretraining framework, called Denoising Autoregressive Predictive Coding (DN-APC), to enhance TS-VAD performance in noisy conditions. We also explore various speaker conditioning methods and evaluate their performance under different noisy conditions. Our experiments show that DN-APC improves performance in noisy conditions, with a general improvement of approx. 2% in both seen and unseen noise. Additionally, we find that FiLM conditioning provides the best overall performance. Representation analysis via tSNE plots reveals robust initial representations of speech and non-speech from pretraining. This underscores the effectiveness of SSL pretraining in improving the robustness and performance of TS-VAD models in noisy environments. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_03184 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Noise-Robust Target-Speaker Voice Activity Detection Through Self-Supervised Pretraining Bovbjerg, Holger Severin Østergaard, Jan Jensen, Jesper Tan, Zheng-Hua Audio and Speech Processing Machine Learning Sound 68T10 I.2.6 Target-Speaker Voice Activity Detection (TS-VAD) is the task of detecting the presence of speech from a known target-speaker in an audio frame. Recently, deep neural network-based models have shown good performance in this task. However, training these models requires extensive labelled data, which is costly and time-consuming to obtain, particularly if generalization to unseen environments is crucial. To mitigate this, we propose a causal, Self-Supervised Learning (SSL) pretraining framework, called Denoising Autoregressive Predictive Coding (DN-APC), to enhance TS-VAD performance in noisy conditions. We also explore various speaker conditioning methods and evaluate their performance under different noisy conditions. Our experiments show that DN-APC improves performance in noisy conditions, with a general improvement of approx. 2% in both seen and unseen noise. Additionally, we find that FiLM conditioning provides the best overall performance. Representation analysis via tSNE plots reveals robust initial representations of speech and non-speech from pretraining. This underscores the effectiveness of SSL pretraining in improving the robustness and performance of TS-VAD models in noisy environments. |
| title | Noise-Robust Target-Speaker Voice Activity Detection Through Self-Supervised Pretraining |
| topic | Audio and Speech Processing Machine Learning Sound 68T10 I.2.6 |
| url | https://arxiv.org/abs/2501.03184 |