Robust Wake Word Spotting With Frame-Level Cross-Modal Attention Based Audio-Visual Conformer
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910352753557504 |
|---|---|
| author | Wang, Haoxu Cheng, Ming Fu, Qiang Li, Ming |
| author_facet | Wang, Haoxu Cheng, Ming Fu, Qiang Li, Ming |
| contents | In recent years, neural network-based Wake Word Spotting achieves good performance on clean audio samples but struggles in noisy environments. Audio-Visual Wake Word Spotting (AVWWS) receives lots of attention because visual lip movement information is not affected by complex acoustic scenes. Previous works usually use simple addition or concatenation for multi-modal fusion. The inter-modal correlation remains relatively under-explored. In this paper, we propose a novel module called Frame-Level Cross-Modal Attention (FLCMA) to improve the performance of AVWWS systems. This module can help model multi-modal information at the frame-level through synchronous lip movements and speech signals. We train the end-to-end FLCMA based Audio-Visual Conformer and further improve the performance by fine-tuning pre-trained uni-modal models for the AVWWS task. The proposed system achieves a new state-of-the-art result (4.57% WWS score) on the far-field MISP dataset. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_01700 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Robust Wake Word Spotting With Frame-Level Cross-Modal Attention Based Audio-Visual Conformer Wang, Haoxu Cheng, Ming Fu, Qiang Li, Ming Sound Multimedia Audio and Speech Processing In recent years, neural network-based Wake Word Spotting achieves good performance on clean audio samples but struggles in noisy environments. Audio-Visual Wake Word Spotting (AVWWS) receives lots of attention because visual lip movement information is not affected by complex acoustic scenes. Previous works usually use simple addition or concatenation for multi-modal fusion. The inter-modal correlation remains relatively under-explored. In this paper, we propose a novel module called Frame-Level Cross-Modal Attention (FLCMA) to improve the performance of AVWWS systems. This module can help model multi-modal information at the frame-level through synchronous lip movements and speech signals. We train the end-to-end FLCMA based Audio-Visual Conformer and further improve the performance by fine-tuning pre-trained uni-modal models for the AVWWS task. The proposed system achieves a new state-of-the-art result (4.57% WWS score) on the far-field MISP dataset. |
| title | Robust Wake Word Spotting With Frame-Level Cross-Modal Attention Based Audio-Visual Conformer |
| topic | Sound Multimedia Audio and Speech Processing |
| url | https://arxiv.org/abs/2403.01700 |