Towards Event-oriented Long Video Understanding
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866929392254451712 |
|---|---|
| author | Du, Yifan Zhou, Kun Huo, Yuqi Li, Yifan Zhao, Wayne Xin Lu, Haoyu Zhao, Zijia Wang, Bingning Chen, Weipeng Wen, Ji-Rong |
| author_facet | Du, Yifan Zhou, Kun Huo, Yuqi Li, Yifan Zhao, Wayne Xin Lu, Haoyu Zhao, Zijia Wang, Bingning Chen, Weipeng Wen, Ji-Rong |
| contents | With the rapid development of video Multimodal Large Language Models (MLLMs), numerous benchmarks have been proposed to assess their video understanding capability. However, due to the lack of rich events in the videos, these datasets may suffer from the short-cut bias that the answers can be deduced from a few frames, without the need to watch the entire video. To address this issue, we introduce Event-Bench, an event-oriented long video understanding benchmark built on existing datasets and human annotations. Event-Bench includes six event-related tasks and 2,190 test instances to comprehensively evaluate video event understanding ability. Additionally, we propose Video Instruction Merging~(VIM), a cost-effective method that enhances video MLLMs using merged, event-intensive video instructions, addressing the scarcity of human-annotated, event-intensive data. Extensive experiments show that the best-performing model, GPT-4o, achieves an overall accuracy of 53.33, significantly outperforming the best open-source model by 41.42%. Leveraging an effective instruction synthesis method and an adaptive model architecture, VIM surpasses both state-of-the-art open-source models and GPT-4V on the Event-Bench. All code, data, and models are publicly available at https://github.com/RUCAIBox/Event-Bench. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_14129 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Towards Event-oriented Long Video Understanding Du, Yifan Zhou, Kun Huo, Yuqi Li, Yifan Zhao, Wayne Xin Lu, Haoyu Zhao, Zijia Wang, Bingning Chen, Weipeng Wen, Ji-Rong Computer Vision and Pattern Recognition Computation and Language Multimedia With the rapid development of video Multimodal Large Language Models (MLLMs), numerous benchmarks have been proposed to assess their video understanding capability. However, due to the lack of rich events in the videos, these datasets may suffer from the short-cut bias that the answers can be deduced from a few frames, without the need to watch the entire video. To address this issue, we introduce Event-Bench, an event-oriented long video understanding benchmark built on existing datasets and human annotations. Event-Bench includes six event-related tasks and 2,190 test instances to comprehensively evaluate video event understanding ability. Additionally, we propose Video Instruction Merging~(VIM), a cost-effective method that enhances video MLLMs using merged, event-intensive video instructions, addressing the scarcity of human-annotated, event-intensive data. Extensive experiments show that the best-performing model, GPT-4o, achieves an overall accuracy of 53.33, significantly outperforming the best open-source model by 41.42%. Leveraging an effective instruction synthesis method and an adaptive model architecture, VIM surpasses both state-of-the-art open-source models and GPT-4V on the Event-Bench. All code, data, and models are publicly available at https://github.com/RUCAIBox/Event-Bench. |
| title | Towards Event-oriented Long Video Understanding |
| topic | Computer Vision and Pattern Recognition Computation and Language Multimedia |
| url | https://arxiv.org/abs/2406.14129 |