Lightweight Wasserstein Audio-Visual Model for Unified Speech Enhancement and Separation
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866915659439407104 |
|---|---|
| author | Park, Jisoo Lee, Seonghak Kim, Guisik Kim, Taewoo Kwon, Junseok |
| author_facet | Park, Jisoo Lee, Seonghak Kim, Guisik Kim, Taewoo Kwon, Junseok |
| contents | Speech Enhancement (SE) and Speech Separation (SS) have traditionally been treated as distinct tasks in speech processing. However, real-world audio often involves both background noise and overlapping speakers, motivating the need for a unified solution. While recent approaches have attempted to integrate SE and SS within multi-stage architectures, these approaches typically involve complex, parameter-heavy models and rely on supervised training, limiting scalability and generalization. In this work, we propose UniVoiceLite, a lightweight and unsupervised audio-visual framework that unifies SE and SS within a single model. UniVoiceLite leverages lip motion and facial identity cues to guide speech extraction and employs Wasserstein distance regularization to stabilize the latent space without requiring paired noisy-clean data. Experimental results demonstrate that UniVoiceLite achieves strong performance in both noisy and multi-speaker scenarios, combining efficiency with robust generalization. The source code is available at https://github.com/jisoo-o/UniVoiceLite. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_06689 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Lightweight Wasserstein Audio-Visual Model for Unified Speech Enhancement and Separation Park, Jisoo Lee, Seonghak Kim, Guisik Kim, Taewoo Kwon, Junseok Computer Vision and Pattern Recognition Audio and Speech Processing Speech Enhancement (SE) and Speech Separation (SS) have traditionally been treated as distinct tasks in speech processing. However, real-world audio often involves both background noise and overlapping speakers, motivating the need for a unified solution. While recent approaches have attempted to integrate SE and SS within multi-stage architectures, these approaches typically involve complex, parameter-heavy models and rely on supervised training, limiting scalability and generalization. In this work, we propose UniVoiceLite, a lightweight and unsupervised audio-visual framework that unifies SE and SS within a single model. UniVoiceLite leverages lip motion and facial identity cues to guide speech extraction and employs Wasserstein distance regularization to stabilize the latent space without requiring paired noisy-clean data. Experimental results demonstrate that UniVoiceLite achieves strong performance in both noisy and multi-speaker scenarios, combining efficiency with robust generalization. The source code is available at https://github.com/jisoo-o/UniVoiceLite. |
| title | Lightweight Wasserstein Audio-Visual Model for Unified Speech Enhancement and Separation |
| topic | Computer Vision and Pattern Recognition Audio and Speech Processing |
| url | https://arxiv.org/abs/2512.06689 |