mWhisper-Flamingo for Multilingual Audio-Visual Noise-Robust Speech Recognition
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915276032835584 |
|---|---|
| author | Rouditchenko, Andrew Thomas, Samuel Kuehne, Hilde Feris, Rogerio Glass, James |
| author_facet | Rouditchenko, Andrew Thomas, Samuel Kuehne, Hilde Feris, Rogerio Glass, James |
| contents | Audio-Visual Speech Recognition (AVSR) combines lip-based video with audio and can improve performance in noise, but most methods are trained only on English data. One limitation is the lack of large-scale multilingual video data, which makes it hard to train models from scratch. In this work, we propose mWhisper-Flamingo for multilingual AVSR which combines the strengths of a pre-trained audio model (Whisper) and video model (AV-HuBERT). To enable better multi-modal integration and improve the noisy multilingual performance, we introduce decoder modality dropout where the model is trained both on paired audio-visual inputs and separate audio/visual inputs. mWhisper-Flamingo achieves state-of-the-art WER on MuAViC, an AVSR dataset of 9 languages. Audio-visual mWhisper-Flamingo consistently outperforms audio-only Whisper on all languages in noisy conditions. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_01547 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | mWhisper-Flamingo for Multilingual Audio-Visual Noise-Robust Speech Recognition Rouditchenko, Andrew Thomas, Samuel Kuehne, Hilde Feris, Rogerio Glass, James Audio and Speech Processing Computer Vision and Pattern Recognition Sound Audio-Visual Speech Recognition (AVSR) combines lip-based video with audio and can improve performance in noise, but most methods are trained only on English data. One limitation is the lack of large-scale multilingual video data, which makes it hard to train models from scratch. In this work, we propose mWhisper-Flamingo for multilingual AVSR which combines the strengths of a pre-trained audio model (Whisper) and video model (AV-HuBERT). To enable better multi-modal integration and improve the noisy multilingual performance, we introduce decoder modality dropout where the model is trained both on paired audio-visual inputs and separate audio/visual inputs. mWhisper-Flamingo achieves state-of-the-art WER on MuAViC, an AVSR dataset of 9 languages. Audio-visual mWhisper-Flamingo consistently outperforms audio-only Whisper on all languages in noisy conditions. |
| title | mWhisper-Flamingo for Multilingual Audio-Visual Noise-Robust Speech Recognition |
| topic | Audio and Speech Processing Computer Vision and Pattern Recognition Sound |
| url | https://arxiv.org/abs/2502.01547 |