Leveraging Mamba with Full-Face Vision for Audio-Visual Speech Enhancement
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866916978560598016 |
|---|---|
| author | Chao, Rong Ren, Wenze Li, You-Jin Hung, Kuo-Hsuan Huang, Sung-Feng Fu, Szu-Wei Cheng, Wen-Huang Tsao, Yu |
| author_facet | Chao, Rong Ren, Wenze Li, You-Jin Hung, Kuo-Hsuan Huang, Sung-Feng Fu, Szu-Wei Cheng, Wen-Huang Tsao, Yu |
| contents | Recent Mamba-based models have shown promise in speech enhancement by efficiently modeling long-range temporal dependencies. However, models like Speech Enhancement Mamba (SEMamba) remain limited to single-speaker scenarios and struggle in complex multi-speaker environments such as the cocktail party problem. To overcome this, we introduce AVSEMamba, an audio-visual speech enhancement model that integrates full-face visual cues with a Mamba-based temporal backbone. By leveraging spatiotemporal visual information, AVSEMamba enables more accurate extraction of target speech in challenging conditions. Evaluated on the AVSEC-4 Challenge development and blind test sets, AVSEMamba outperforms other monaural baselines in speech intelligibility (STOI), perceptual quality (PESQ), and non-intrusive quality (UTMOS), and achieves \textbf{1st place} on the monaural leaderboard. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_13624 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Leveraging Mamba with Full-Face Vision for Audio-Visual Speech Enhancement Chao, Rong Ren, Wenze Li, You-Jin Hung, Kuo-Hsuan Huang, Sung-Feng Fu, Szu-Wei Cheng, Wen-Huang Tsao, Yu Sound Audio and Speech Processing Recent Mamba-based models have shown promise in speech enhancement by efficiently modeling long-range temporal dependencies. However, models like Speech Enhancement Mamba (SEMamba) remain limited to single-speaker scenarios and struggle in complex multi-speaker environments such as the cocktail party problem. To overcome this, we introduce AVSEMamba, an audio-visual speech enhancement model that integrates full-face visual cues with a Mamba-based temporal backbone. By leveraging spatiotemporal visual information, AVSEMamba enables more accurate extraction of target speech in challenging conditions. Evaluated on the AVSEC-4 Challenge development and blind test sets, AVSEMamba outperforms other monaural baselines in speech intelligibility (STOI), perceptual quality (PESQ), and non-intrusive quality (UTMOS), and achieves \textbf{1st place} on the monaural leaderboard. |
| title | Leveraging Mamba with Full-Face Vision for Audio-Visual Speech Enhancement |
| topic | Sound Audio and Speech Processing |
| url | https://arxiv.org/abs/2508.13624 |