Extract and Diffuse: Latent Integration for Improved Diffusion-based Speech and Vocal Enhancement
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866918145167458304 |
|---|---|
| author | Yang, Yudong Liu, Zhan Yu, Wenyi Sun, Guangzhi Kong, Qiuqiang Zhang, Chao |
| author_facet | Yang, Yudong Liu, Zhan Yu, Wenyi Sun, Guangzhi Kong, Qiuqiang Zhang, Chao |
| contents | Diffusion-based generative models have recently achieved remarkable results in speech and vocal enhancement due to their ability to model complex speech data distributions. While these models generalize well to unseen acoustic environments, they may not achieve the same level of fidelity as the discriminative models specifically trained to enhance particular acoustic conditions. In this paper, we propose Ex-Diff, a novel score-based diffusion model that integrates the latent representations produced by a discriminative model to improve speech and vocal enhancement, which combines the strengths of both generative and discriminative models. Experimental results on the widely used MUSDB dataset show relative improvements of 3.7% in SI-SDR and 10.0% in SI-SIR compared to the baseline diffusion model for speech and vocal enhancement tasks, respectively. Additionally, case studies are provided to further illustrate and analyze the complementary nature of generative and discriminative models in this context. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_09642 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Extract and Diffuse: Latent Integration for Improved Diffusion-based Speech and Vocal Enhancement Yang, Yudong Liu, Zhan Yu, Wenyi Sun, Guangzhi Kong, Qiuqiang Zhang, Chao Audio and Speech Processing Machine Learning Sound Diffusion-based generative models have recently achieved remarkable results in speech and vocal enhancement due to their ability to model complex speech data distributions. While these models generalize well to unseen acoustic environments, they may not achieve the same level of fidelity as the discriminative models specifically trained to enhance particular acoustic conditions. In this paper, we propose Ex-Diff, a novel score-based diffusion model that integrates the latent representations produced by a discriminative model to improve speech and vocal enhancement, which combines the strengths of both generative and discriminative models. Experimental results on the widely used MUSDB dataset show relative improvements of 3.7% in SI-SDR and 10.0% in SI-SIR compared to the baseline diffusion model for speech and vocal enhancement tasks, respectively. Additionally, case studies are provided to further illustrate and analyze the complementary nature of generative and discriminative models in this context. |
| title | Extract and Diffuse: Latent Integration for Improved Diffusion-based Speech and Vocal Enhancement |
| topic | Audio and Speech Processing Machine Learning Sound |
| url | https://arxiv.org/abs/2409.09642 |