Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2503.00162 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929736074133504 |
|---|---|
| author | Wei, Kangda Zhou, Zhengyu Wang, Bingqing Araki, Jun Lange, Lukas Huang, Ruihong Feng, Zhe |
| author_facet | Wei, Kangda Zhou, Zhengyu Wang, Bingqing Araki, Jun Lange, Lukas Huang, Ruihong Feng, Zhe |
| contents | In recent years, online lecture videos have become an increasingly popular resource for acquiring new knowledge. Systems capable of effectively understanding/indexing lecture videos are thus highly desirable, enabling downstream tasks like question answering to help users efficiently locate specific information within videos. This work proposes PreMind, a novel multi-agent multimodal framework that leverages various large models for advanced understanding/indexing of presentation-style videos. PreMind first segments videos into slide-presentation segments using a Vision-Language Model (VLM) to enhance modern shot-detection techniques. Each segment is then analyzed to generate multimodal indexes through three key steps: (1) extracting slide visual content, (2) transcribing speech narratives, and (3) consolidating these visual and speech contents into an integrated understanding. Three innovative mechanisms are also proposed to improve performance: leveraging prior lecture knowledge to refine visual understanding, detecting/correcting speech transcription errors using a VLM, and utilizing a critic agent for dynamic iterative self-reflection in vision analysis. Compared to traditional video indexing methods, PreMind captures rich, reliable multimodal information, allowing users to search for details like abbreviations shown only on slides. Systematic evaluations on the public LPM dataset and an internal enterprise dataset are conducted to validate PreMind's effectiveness, supported by detailed analyses. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_00162 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | PreMind: Multi-Agent Video Understanding for Advanced Indexing of Presentation-style Videos Wei, Kangda Zhou, Zhengyu Wang, Bingqing Araki, Jun Lange, Lukas Huang, Ruihong Feng, Zhe Computer Vision and Pattern Recognition Artificial Intelligence Computation and Language Multiagent Systems In recent years, online lecture videos have become an increasingly popular resource for acquiring new knowledge. Systems capable of effectively understanding/indexing lecture videos are thus highly desirable, enabling downstream tasks like question answering to help users efficiently locate specific information within videos. This work proposes PreMind, a novel multi-agent multimodal framework that leverages various large models for advanced understanding/indexing of presentation-style videos. PreMind first segments videos into slide-presentation segments using a Vision-Language Model (VLM) to enhance modern shot-detection techniques. Each segment is then analyzed to generate multimodal indexes through three key steps: (1) extracting slide visual content, (2) transcribing speech narratives, and (3) consolidating these visual and speech contents into an integrated understanding. Three innovative mechanisms are also proposed to improve performance: leveraging prior lecture knowledge to refine visual understanding, detecting/correcting speech transcription errors using a VLM, and utilizing a critic agent for dynamic iterative self-reflection in vision analysis. Compared to traditional video indexing methods, PreMind captures rich, reliable multimodal information, allowing users to search for details like abbreviations shown only on slides. Systematic evaluations on the public LPM dataset and an internal enterprise dataset are conducted to validate PreMind's effectiveness, supported by detailed analyses. |
| title | PreMind: Multi-Agent Video Understanding for Advanced Indexing of Presentation-style Videos |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Computation and Language Multiagent Systems |
| url | https://arxiv.org/abs/2503.00162 |