Audio-Visual Feature Synchronization for Robust Speech Enhancement in Hearing Aids
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866915465971892224 |
|---|---|
| author | Saleem, Nasir Gogate, Mandar Dashtipour, Kia Hussain, Adeel Anwar, Usman Adetomi, Adewale Arslan, Tughrul Hussain, Amir |
| author_facet | Saleem, Nasir Gogate, Mandar Dashtipour, Kia Hussain, Adeel Anwar, Usman Adetomi, Adewale Arslan, Tughrul Hussain, Amir |
| contents | Audio-visual feature synchronization for real-time speech enhancement in hearing aids represents a progressive approach to improving speech intelligibility and user experience, particularly in strong noisy backgrounds. This approach integrates auditory signals with visual cues, utilizing the complementary description of these modalities to improve speech intelligibility. Audio-visual feature synchronization for real-time SE in hearing aids can be further optimized using an efficient feature alignment module. In this study, a lightweight cross-attentional model learns robust audio-visual representations by exploiting large-scale data and simple architecture. By incorporating the lightweight cross-attentional model in an AVSE framework, the neural system dynamically emphasizes critical features across audio and visual modalities, enabling defined synchronization and improved speech intelligibility. The proposed AVSE model not only ensures high performance in noise suppression and feature alignment but also achieves real-time processing with minimal latency (36ms) and energy consumption. Evaluations on the AVSEC3 dataset show the efficiency of the model, achieving significant gains over baselines in perceptual quality (PESQ:0.52), intelligibility (STOI:19\%), and fidelity (SI-SDR:10.10dB). |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_19483 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Audio-Visual Feature Synchronization for Robust Speech Enhancement in Hearing Aids Saleem, Nasir Gogate, Mandar Dashtipour, Kia Hussain, Adeel Anwar, Usman Adetomi, Adewale Arslan, Tughrul Hussain, Amir Audio and Speech Processing Audio-visual feature synchronization for real-time speech enhancement in hearing aids represents a progressive approach to improving speech intelligibility and user experience, particularly in strong noisy backgrounds. This approach integrates auditory signals with visual cues, utilizing the complementary description of these modalities to improve speech intelligibility. Audio-visual feature synchronization for real-time SE in hearing aids can be further optimized using an efficient feature alignment module. In this study, a lightweight cross-attentional model learns robust audio-visual representations by exploiting large-scale data and simple architecture. By incorporating the lightweight cross-attentional model in an AVSE framework, the neural system dynamically emphasizes critical features across audio and visual modalities, enabling defined synchronization and improved speech intelligibility. The proposed AVSE model not only ensures high performance in noise suppression and feature alignment but also achieves real-time processing with minimal latency (36ms) and energy consumption. Evaluations on the AVSEC3 dataset show the efficiency of the model, achieving significant gains over baselines in perceptual quality (PESQ:0.52), intelligibility (STOI:19\%), and fidelity (SI-SDR:10.10dB). |
| title | Audio-Visual Feature Synchronization for Robust Speech Enhancement in Hearing Aids |
| topic | Audio and Speech Processing |
| url | https://arxiv.org/abs/2508.19483 |