Exploring the Potential of Multimodal LLM with Knowledge-Intensive Multimodal ASR
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866915018764713984 |
|---|---|
| author | Wang, Minghan Wang, Yuxia Vu, Thuy-Trang Shareghi, Ehsan Haffari, Gholamreza |
| author_facet | Wang, Minghan Wang, Yuxia Vu, Thuy-Trang Shareghi, Ehsan Haffari, Gholamreza |
| contents | Recent advancements in multimodal large language models (MLLMs) have made significant progress in integrating information across various modalities, yet real-world applications in educational and scientific domains remain challenging. This paper introduces the Multimodal Scientific ASR (MS-ASR) task, which focuses on transcribing scientific conference videos by leveraging visual information from slides to enhance the accuracy of technical terminologies. Realized that traditional metrics like WER fall short in assessing performance accurately, prompting the proposal of severity-aware WER (SWER) that considers the content type and severity of ASR errors. We propose the Scientific Vision Augmented ASR (SciVASR) framework as a baseline method, enabling MLLMs to improve transcript quality through post-editing. Evaluations of state-of-the-art MLLMs, including GPT-4o, show a 45% improvement over speech-only baselines, highlighting the importance of multimodal information integration. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_10880 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Exploring the Potential of Multimodal LLM with Knowledge-Intensive Multimodal ASR Wang, Minghan Wang, Yuxia Vu, Thuy-Trang Shareghi, Ehsan Haffari, Gholamreza Computation and Language Recent advancements in multimodal large language models (MLLMs) have made significant progress in integrating information across various modalities, yet real-world applications in educational and scientific domains remain challenging. This paper introduces the Multimodal Scientific ASR (MS-ASR) task, which focuses on transcribing scientific conference videos by leveraging visual information from slides to enhance the accuracy of technical terminologies. Realized that traditional metrics like WER fall short in assessing performance accurately, prompting the proposal of severity-aware WER (SWER) that considers the content type and severity of ASR errors. We propose the Scientific Vision Augmented ASR (SciVASR) framework as a baseline method, enabling MLLMs to improve transcript quality through post-editing. Evaluations of state-of-the-art MLLMs, including GPT-4o, show a 45% improvement over speech-only baselines, highlighting the importance of multimodal information integration. |
| title | Exploring the Potential of Multimodal LLM with Knowledge-Intensive Multimodal ASR |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2406.10880 |