Real-Time Audio-Visual Speech Enhancement Using Pre-trained Visual Representations
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866913974029647872 |
|---|---|
| author | Ma, T. Aleksandra Yin, Sile Yang, Li-Chia Zhang, Shuo |
| author_facet | Ma, T. Aleksandra Yin, Sile Yang, Li-Chia Zhang, Shuo |
| contents | Speech enhancement in audio-only settings remains challenging, particularly in the presence of interfering speakers. This paper presents a simple yet effective real-time audio-visual speech enhancement (AVSE) system, RAVEN, which isolates and enhances the on-screen target speaker while suppressing interfering speakers and background noise. We investigate how visual embeddings learned from audio-visual speech recognition (AVSR) and active speaker detection (ASD) contribute to AVSE across different SNR conditions and numbers of interfering speakers. Our results show concatenating embeddings from AVSR and ASD models provides the greatest improvement in low-SNR, multi-speaker environments, while AVSR embeddings alone perform best in noise-only scenarios. In addition, we develop a real-time streaming system that operates on a computer CPU and we provide a video demonstration and code repository. To our knowledge, this is the first open-source implementation of a real-time AVSE system. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_21448 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Real-Time Audio-Visual Speech Enhancement Using Pre-trained Visual Representations Ma, T. Aleksandra Yin, Sile Yang, Li-Chia Zhang, Shuo Audio and Speech Processing Emerging Technologies Machine Learning Speech enhancement in audio-only settings remains challenging, particularly in the presence of interfering speakers. This paper presents a simple yet effective real-time audio-visual speech enhancement (AVSE) system, RAVEN, which isolates and enhances the on-screen target speaker while suppressing interfering speakers and background noise. We investigate how visual embeddings learned from audio-visual speech recognition (AVSR) and active speaker detection (ASD) contribute to AVSE across different SNR conditions and numbers of interfering speakers. Our results show concatenating embeddings from AVSR and ASD models provides the greatest improvement in low-SNR, multi-speaker environments, while AVSR embeddings alone perform best in noise-only scenarios. In addition, we develop a real-time streaming system that operates on a computer CPU and we provide a video demonstration and code repository. To our knowledge, this is the first open-source implementation of a real-time AVSE system. |
| title | Real-Time Audio-Visual Speech Enhancement Using Pre-trained Visual Representations |
| topic | Audio and Speech Processing Emerging Technologies Machine Learning |
| url | https://arxiv.org/abs/2507.21448 |