Audio-Visual Speech Enhancement for Spatial Audio - Spatial-VisualVoice and the MAVE Database
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866912657641046016 |
|---|---|
| author | Yaffe, Danielle Campe, Ferdinand Sharma, Prachi Kolossa, Dorothea Rafaely, Boaz |
| author_facet | Yaffe, Danielle Campe, Ferdinand Sharma, Prachi Kolossa, Dorothea Rafaely, Boaz |
| contents | Audio-visual speech enhancement (AVSE) has been found to be particularly useful at low signal-to-noise (SNR) ratios due to the immunity of the visual features to acoustic noise. However, a significant gap exists in AVSE methods tailored to enhance spatial audio under low-SNR conditions. The latter is of growing interest with augmented reality applications. To address this gap, we present a multi-channel AVSE framework based on VisualVoice that leverages spatial cues from microphone arrays and visual information for enhancing the target speaker in noisy environments. We also introduce MAVe, a novel database containing multi-channel audio-visual signals in controlled, reproducible room conditions across a wide range of SNR levels. Experiments demonstrate that the proposed method consistently achieves significant gains in SI-SDR, STOI, and PESQ, particularly in low SNRs. Binaural signal analysis further confirms the preservation of spatial cues and intelligibility. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_16437 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Audio-Visual Speech Enhancement for Spatial Audio - Spatial-VisualVoice and the MAVE Database Yaffe, Danielle Campe, Ferdinand Sharma, Prachi Kolossa, Dorothea Rafaely, Boaz Audio and Speech Processing Audio-visual speech enhancement (AVSE) has been found to be particularly useful at low signal-to-noise (SNR) ratios due to the immunity of the visual features to acoustic noise. However, a significant gap exists in AVSE methods tailored to enhance spatial audio under low-SNR conditions. The latter is of growing interest with augmented reality applications. To address this gap, we present a multi-channel AVSE framework based on VisualVoice that leverages spatial cues from microphone arrays and visual information for enhancing the target speaker in noisy environments. We also introduce MAVe, a novel database containing multi-channel audio-visual signals in controlled, reproducible room conditions across a wide range of SNR levels. Experiments demonstrate that the proposed method consistently achieves significant gains in SI-SDR, STOI, and PESQ, particularly in low SNRs. Binaural signal analysis further confirms the preservation of spatial cues and intelligibility. |
| title | Audio-Visual Speech Enhancement for Spatial Audio - Spatial-VisualVoice and the MAVE Database |
| topic | Audio and Speech Processing |
| url | https://arxiv.org/abs/2510.16437 |