Salvato in:
| Autori principali: | , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | https://arxiv.org/abs/2410.05301 |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866913650680266752 |
|---|---|
| author | Ayilo, Jean-Eudes Sadeghi, Mostafa Serizel, Romain Alameda-Pineda, Xavier |
| author_facet | Ayilo, Jean-Eudes Sadeghi, Mostafa Serizel, Romain Alameda-Pineda, Xavier |
| contents | This paper proposes a new unsupervised audio-visual speech enhancement (AVSE) approach that combines a diffusion-based audio-visual speech generative model with a non-negative matrix factorization (NMF) noise model. First, the diffusion model is pre-trained on clean speech conditioned on corresponding video data to simulate the speech generative distribution. This pre-trained model is then paired with the NMF-based noise model to estimate clean speech iteratively. Specifically, a diffusion-based posterior sampling approach is implemented within the reverse diffusion process, where after each iteration, a speech estimate is obtained and used to update the noise parameters. Experimental results confirm that the proposed AVSE approach not only outperforms its audio-only counterpart but also generalizes better than a recent supervised-generative AVSE method. Additionally, the new inference algorithm offers a better balance between inference speed and performance compared to the previous diffusion-based method. Code and demo available at: https://jeaneudesayilo.github.io/fast_UdiffSE |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_05301 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Diffusion-based Unsupervised Audio-visual Speech Enhancement Ayilo, Jean-Eudes Sadeghi, Mostafa Serizel, Romain Alameda-Pineda, Xavier Sound Artificial Intelligence Computer Vision and Pattern Recognition Machine Learning Audio and Speech Processing Signal Processing This paper proposes a new unsupervised audio-visual speech enhancement (AVSE) approach that combines a diffusion-based audio-visual speech generative model with a non-negative matrix factorization (NMF) noise model. First, the diffusion model is pre-trained on clean speech conditioned on corresponding video data to simulate the speech generative distribution. This pre-trained model is then paired with the NMF-based noise model to estimate clean speech iteratively. Specifically, a diffusion-based posterior sampling approach is implemented within the reverse diffusion process, where after each iteration, a speech estimate is obtained and used to update the noise parameters. Experimental results confirm that the proposed AVSE approach not only outperforms its audio-only counterpart but also generalizes better than a recent supervised-generative AVSE method. Additionally, the new inference algorithm offers a better balance between inference speed and performance compared to the previous diffusion-based method. Code and demo available at: https://jeaneudesayilo.github.io/fast_UdiffSE |
| title | Diffusion-based Unsupervised Audio-visual Speech Enhancement |
| topic | Sound Artificial Intelligence Computer Vision and Pattern Recognition Machine Learning Audio and Speech Processing Signal Processing |
| url | https://arxiv.org/abs/2410.05301 |