Zero-Shot Long-Form Video Understanding through Screenplay
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866913403641004032 |
|---|---|
| author | Wu, Yongliang Li, Bozheng Cao, Jiawang Zhu, Wenbo Lu, Yi Chi, Weiheng Xie, Chuyun Zheng, Haolin Su, Ziyue Wu, Jay Yang, Xu |
| author_facet | Wu, Yongliang Li, Bozheng Cao, Jiawang Zhu, Wenbo Lu, Yi Chi, Weiheng Xie, Chuyun Zheng, Haolin Su, Ziyue Wu, Jay Yang, Xu |
| contents | The Long-form Video Question-Answering task requires the comprehension and analysis of extended video content to respond accurately to questions by utilizing both temporal and contextual information. In this paper, we present MM-Screenplayer, an advanced video understanding system with multi-modal perception capabilities that can convert any video into textual screenplay representations. Unlike previous storytelling methods, we organize video content into scenes as the basic unit, rather than just visually continuous shots. Additionally, we developed a ``Look Back'' strategy to reassess and validate uncertain information, particularly targeting breakpoint mode. MM-Screenplayer achieved highest score in the CVPR'2024 LOng-form VidEo Understanding (LOVEU) Track 1 Challenge, with a global accuracy of 87.5% and a breakpoint accuracy of 68.8%. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_17309 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Zero-Shot Long-Form Video Understanding through Screenplay Wu, Yongliang Li, Bozheng Cao, Jiawang Zhu, Wenbo Lu, Yi Chi, Weiheng Xie, Chuyun Zheng, Haolin Su, Ziyue Wu, Jay Yang, Xu Computer Vision and Pattern Recognition The Long-form Video Question-Answering task requires the comprehension and analysis of extended video content to respond accurately to questions by utilizing both temporal and contextual information. In this paper, we present MM-Screenplayer, an advanced video understanding system with multi-modal perception capabilities that can convert any video into textual screenplay representations. Unlike previous storytelling methods, we organize video content into scenes as the basic unit, rather than just visually continuous shots. Additionally, we developed a ``Look Back'' strategy to reassess and validate uncertain information, particularly targeting breakpoint mode. MM-Screenplayer achieved highest score in the CVPR'2024 LOng-form VidEo Understanding (LOVEU) Track 1 Challenge, with a global accuracy of 87.5% and a breakpoint accuracy of 68.8%. |
| title | Zero-Shot Long-Form Video Understanding through Screenplay |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2406.17309 |