SlideAVSR: A Dataset of Paper Explanation Videos for Audio-Visual Speech Recognition
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909237416820736 |
|---|---|
| author | Wang, Hao Kurita, Shuhei Shimizu, Shuichiro Kawahara, Daisuke |
| author_facet | Wang, Hao Kurita, Shuhei Shimizu, Shuichiro Kawahara, Daisuke |
| contents | Audio-visual speech recognition (AVSR) is a multimodal extension of automatic speech recognition (ASR), using video as a complement to audio. In AVSR, considerable efforts have been directed at datasets for facial features such as lip-readings, while they often fall short in evaluating the image comprehension capabilities in broader contexts. In this paper, we construct SlideAVSR, an AVSR dataset using scientific paper explanation videos. SlideAVSR provides a new benchmark where models transcribe speech utterances with texts on the slides on the presentation recordings. As technical terminologies that are frequent in paper explanations are notoriously challenging to transcribe without reference texts, our SlideAVSR dataset spotlights a new aspect of AVSR problems. As a simple yet effective baseline, we propose DocWhisper, an AVSR model that can refer to textual information from slides, and confirm its effectiveness on SlideAVSR. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2401_09759 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | SlideAVSR: A Dataset of Paper Explanation Videos for Audio-Visual Speech Recognition Wang, Hao Kurita, Shuhei Shimizu, Shuichiro Kawahara, Daisuke Computer Vision and Pattern Recognition Multimedia Sound Audio and Speech Processing Audio-visual speech recognition (AVSR) is a multimodal extension of automatic speech recognition (ASR), using video as a complement to audio. In AVSR, considerable efforts have been directed at datasets for facial features such as lip-readings, while they often fall short in evaluating the image comprehension capabilities in broader contexts. In this paper, we construct SlideAVSR, an AVSR dataset using scientific paper explanation videos. SlideAVSR provides a new benchmark where models transcribe speech utterances with texts on the slides on the presentation recordings. As technical terminologies that are frequent in paper explanations are notoriously challenging to transcribe without reference texts, our SlideAVSR dataset spotlights a new aspect of AVSR problems. As a simple yet effective baseline, we propose DocWhisper, an AVSR model that can refer to textual information from slides, and confirm its effectiveness on SlideAVSR. |
| title | SlideAVSR: A Dataset of Paper Explanation Videos for Audio-Visual Speech Recognition |
| topic | Computer Vision and Pattern Recognition Multimedia Sound Audio and Speech Processing |
| url | https://arxiv.org/abs/2401.09759 |