Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2406.06964 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909221067423744 |
|---|---|
| author | Mohapatra, Payal Likhite, Shamika Biswas, Subrata Islam, Bashima Zhu, Qi |
| author_facet | Mohapatra, Payal Likhite, Shamika Biswas, Subrata Islam, Bashima Zhu, Qi |
| contents | Most existing speech disfluency detection techniques only rely upon acoustic data. In this work, we present a practical multimodal disfluency detection approach that leverages available video data together with audio. We curate an audiovisual dataset and propose a novel fusion technique with unified weight-sharing modality-agnostic encoders to learn the temporal and semantic context. Our resilient design accommodates real-world scenarios where the video modality may sometimes be missing during inference. We also present alternative fusion strategies when both modalities are assured to be complete. In experiments across five disfluency-detection tasks, our unified multimodal approach significantly outperforms Audio-only unimodal methods, yielding an average absolute improvement of 10% (i.e., 10 percentage point increase) when both video and audio modalities are always available, and 7% even when video modality is missing in half of the samples. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_06964 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Missingness-resilient Video-enhanced Multimodal Disfluency Detection Mohapatra, Payal Likhite, Shamika Biswas, Subrata Islam, Bashima Zhu, Qi Computation and Language Multimedia Sound Audio and Speech Processing Most existing speech disfluency detection techniques only rely upon acoustic data. In this work, we present a practical multimodal disfluency detection approach that leverages available video data together with audio. We curate an audiovisual dataset and propose a novel fusion technique with unified weight-sharing modality-agnostic encoders to learn the temporal and semantic context. Our resilient design accommodates real-world scenarios where the video modality may sometimes be missing during inference. We also present alternative fusion strategies when both modalities are assured to be complete. In experiments across five disfluency-detection tasks, our unified multimodal approach significantly outperforms Audio-only unimodal methods, yielding an average absolute improvement of 10% (i.e., 10 percentage point increase) when both video and audio modalities are always available, and 7% even when video modality is missing in half of the samples. |
| title | Missingness-resilient Video-enhanced Multimodal Disfluency Detection |
| topic | Computation and Language Multimedia Sound Audio and Speech Processing |
| url | https://arxiv.org/abs/2406.06964 |