Seeing the Context: Rich Visual Context-Aware Speech Recognition via Multimodal Reasoning
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2026
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866917321738551296 |
|---|---|
| author | Tian, Wenjie Shao, Mingchen Mu, Bingshen Geng, Xuelong Wang, Chengyou Liao, Yujie Zhao, Zhixian Zhang, Ziyu Hu, Jingbin Wei, Mengqi Xie, Lei |
| author_facet | Tian, Wenjie Shao, Mingchen Mu, Bingshen Geng, Xuelong Wang, Chengyou Liao, Yujie Zhao, Zhixian Zhang, Ziyu Hu, Jingbin Wei, Mengqi Xie, Lei |
| contents | Audio-visual speech recognition (AVSR) is an extension of ASR that incorporates visual signals. Current AVSR approaches primarily focus on lip motion, largely overlooking rich context present in the video such as speaking scene and on-screen text. To tackle such CAVSR (AVSR including rich visual Context), we propose VASR designed to "see" and reason the visual context to improve speech recognition. Specifically, we construct an Audio-Visual Chain-of-Thought (AV-CoT) that explicitly enforces intermediate cross-modal grounding between acoustic signals and visual evidence. This evidence-driven reasoning mitigates the "single-modality dominance" problem, where models either over-rely on visual context or fail to utilize it. Besides, to address the data scarcity, we construct and release a corresponding data pipeline and test set. Experiments show that AV-CoT effectively mitigates the single-modality dominance, achieving state-of-the-art performance in CAVSR. The project is open-sourced. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_07263 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Seeing the Context: Rich Visual Context-Aware Speech Recognition via Multimodal Reasoning Tian, Wenjie Shao, Mingchen Mu, Bingshen Geng, Xuelong Wang, Chengyou Liao, Yujie Zhao, Zhixian Zhang, Ziyu Hu, Jingbin Wei, Mengqi Xie, Lei Sound Audio and Speech Processing Audio-visual speech recognition (AVSR) is an extension of ASR that incorporates visual signals. Current AVSR approaches primarily focus on lip motion, largely overlooking rich context present in the video such as speaking scene and on-screen text. To tackle such CAVSR (AVSR including rich visual Context), we propose VASR designed to "see" and reason the visual context to improve speech recognition. Specifically, we construct an Audio-Visual Chain-of-Thought (AV-CoT) that explicitly enforces intermediate cross-modal grounding between acoustic signals and visual evidence. This evidence-driven reasoning mitigates the "single-modality dominance" problem, where models either over-rely on visual context or fail to utilize it. Besides, to address the data scarcity, we construct and release a corresponding data pipeline and test set. Experiments show that AV-CoT effectively mitigates the single-modality dominance, achieving state-of-the-art performance in CAVSR. The project is open-sourced. |
| title | Seeing the Context: Rich Visual Context-Aware Speech Recognition via Multimodal Reasoning |
| topic | Sound Audio and Speech Processing |
| url | https://arxiv.org/abs/2603.07263 |