BOOM: Beyond Only One Modality KIT's Multimodal Multilingual Lecture Companion
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908845406683136 |
|---|---|
| author | Koneru, Sai Retkowski, Fabian Huber, Christian Hilgert, Lukas Akti, Seymanur Ugan, Enes Yavuz Waibel, Alexander Niehues, Jan |
| author_facet | Koneru, Sai Retkowski, Fabian Huber, Christian Hilgert, Lukas Akti, Seymanur Ugan, Enes Yavuz Waibel, Alexander Niehues, Jan |
| contents | The globalization of education and rapid growth of online learning have made localizing educational content a critical challenge. Lecture materials are inherently multimodal, combining spoken audio with visual slides, which requires systems capable of processing multiple input modalities. To provide an accessible and complete learning experience, translations must preserve all modalities: text for reading, slides for visual understanding, and speech for auditory learning. We present \textbf{BOOM}, a multimodal multilingual lecture companion that jointly translates lecture audio and slides to produce synchronized outputs across three modalities: translated text, localized slides with preserved visual elements, and synthesized speech. This end-to-end approach enables students to access lectures in their native language while aiming to preserve the original content in its entirety. Our experiments demonstrate that slide-aware transcripts also yield cascading benefits for downstream tasks such as summarization and question answering. The demo video and code can be found at https://ai4lt.github.io/boom/ \footnote{All released code and models are licensed under the MIT License}. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_02817 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | BOOM: Beyond Only One Modality KIT's Multimodal Multilingual Lecture Companion Koneru, Sai Retkowski, Fabian Huber, Christian Hilgert, Lukas Akti, Seymanur Ugan, Enes Yavuz Waibel, Alexander Niehues, Jan Computation and Language The globalization of education and rapid growth of online learning have made localizing educational content a critical challenge. Lecture materials are inherently multimodal, combining spoken audio with visual slides, which requires systems capable of processing multiple input modalities. To provide an accessible and complete learning experience, translations must preserve all modalities: text for reading, slides for visual understanding, and speech for auditory learning. We present \textbf{BOOM}, a multimodal multilingual lecture companion that jointly translates lecture audio and slides to produce synchronized outputs across three modalities: translated text, localized slides with preserved visual elements, and synthesized speech. This end-to-end approach enables students to access lectures in their native language while aiming to preserve the original content in its entirety. Our experiments demonstrate that slide-aware transcripts also yield cascading benefits for downstream tasks such as summarization and question answering. The demo video and code can be found at https://ai4lt.github.io/boom/ \footnote{All released code and models are licensed under the MIT License}. |
| title | BOOM: Beyond Only One Modality KIT's Multimodal Multilingual Lecture Companion |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2512.02817 |