Robust Audiovisual Speech Recognition Models with Mixture-of-Experts
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866913506998091776 |
|---|---|
| author | Wu, Yihan Peng, Yifan Lu, Yichen Chang, Xuankai Song, Ruihua Watanabe, Shinji |
| author_facet | Wu, Yihan Peng, Yifan Lu, Yichen Chang, Xuankai Song, Ruihua Watanabe, Shinji |
| contents | Visual signals can enhance audiovisual speech recognition accuracy by providing additional contextual information. Given the complexity of visual signals, an audiovisual speech recognition model requires robust generalization capabilities across diverse video scenarios, presenting a significant challenge. In this paper, we introduce EVA, leveraging the mixture-of-Experts for audioVisual ASR to perform robust speech recognition for ``in-the-wild'' videos. Specifically, we first encode visual information into visual tokens sequence and map them into speech space by a lightweight projection. Then, we build EVA upon a robust pretrained speech recognition model, ensuring its generalization ability. Moreover, to incorporate visual information effectively, we inject visual information into the ASR model through a mixture-of-experts module. Experiments show our model achieves state-of-the-art results on three benchmarks, which demonstrates the generalization ability of EVA across diverse video domains. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_12370 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Robust Audiovisual Speech Recognition Models with Mixture-of-Experts Wu, Yihan Peng, Yifan Lu, Yichen Chang, Xuankai Song, Ruihua Watanabe, Shinji Audio and Speech Processing Computation and Language Computer Vision and Pattern Recognition Sound Visual signals can enhance audiovisual speech recognition accuracy by providing additional contextual information. Given the complexity of visual signals, an audiovisual speech recognition model requires robust generalization capabilities across diverse video scenarios, presenting a significant challenge. In this paper, we introduce EVA, leveraging the mixture-of-Experts for audioVisual ASR to perform robust speech recognition for ``in-the-wild'' videos. Specifically, we first encode visual information into visual tokens sequence and map them into speech space by a lightweight projection. Then, we build EVA upon a robust pretrained speech recognition model, ensuring its generalization ability. Moreover, to incorporate visual information effectively, we inject visual information into the ASR model through a mixture-of-experts module. Experiments show our model achieves state-of-the-art results on three benchmarks, which demonstrates the generalization ability of EVA across diverse video domains. |
| title | Robust Audiovisual Speech Recognition Models with Mixture-of-Experts |
| topic | Audio and Speech Processing Computation and Language Computer Vision and Pattern Recognition Sound |
| url | https://arxiv.org/abs/2409.12370 |