SSNAPS: Audio-Visual Separation of Speech and Background Noise with Diffusion Inverse Sampling
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2026
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866912867014410240 |
|---|---|
| author | Yemini, Yochai Ellinson, Yoav Ben-Ari, Rami Gannot, Sharon Fetaya, Ethan |
| author_facet | Yemini, Yochai Ellinson, Yoav Ben-Ari, Rami Gannot, Sharon Fetaya, Ethan |
| contents | This paper addresses the challenge of audio-visual single-microphone speech separation and enhancement in the presence of real-world environmental noise. Our approach is based on generative inverse sampling, where we model clean speech and ambient noise with dedicated diffusion priors and jointly leverage them to recover all underlying sources. To achieve this, we reformulate a recent inverse sampler to match our setting. We evaluate on mixtures of 1, 2, and 3 speakers with noise and show that, despite being entirely unsupervised, our method consistently outperforms leading supervised baselines in \ac{WER} across all conditions. We further extend our framework to handle off-screen speaker separation. Moreover, the high fidelity of the separated noise component makes it suitable for downstream acoustic scene detection. Demo page: https://ssnapsicml.github.io/ssnapsicml2026/ |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_01394 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | SSNAPS: Audio-Visual Separation of Speech and Background Noise with Diffusion Inverse Sampling Yemini, Yochai Ellinson, Yoav Ben-Ari, Rami Gannot, Sharon Fetaya, Ethan Audio and Speech Processing Machine Learning Sound This paper addresses the challenge of audio-visual single-microphone speech separation and enhancement in the presence of real-world environmental noise. Our approach is based on generative inverse sampling, where we model clean speech and ambient noise with dedicated diffusion priors and jointly leverage them to recover all underlying sources. To achieve this, we reformulate a recent inverse sampler to match our setting. We evaluate on mixtures of 1, 2, and 3 speakers with noise and show that, despite being entirely unsupervised, our method consistently outperforms leading supervised baselines in \ac{WER} across all conditions. We further extend our framework to handle off-screen speaker separation. Moreover, the high fidelity of the separated noise component makes it suitable for downstream acoustic scene detection. Demo page: https://ssnapsicml.github.io/ssnapsicml2026/ |
| title | SSNAPS: Audio-Visual Separation of Speech and Background Noise with Diffusion Inverse Sampling |
| topic | Audio and Speech Processing Machine Learning Sound |
| url | https://arxiv.org/abs/2602.01394 |