VidComposition: Can MLLMs Analyze Compositions in Compiled Videos?
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917101579534336 |
|---|---|
| author | Tang, Yolo Y. Guo, Junjia Hua, Hang Liang, Susan Feng, Mingqian Li, Xinyang Mao, Rui Huang, Chao Bi, Jing Zhang, Zeliang Fazli, Pooyan Xu, Chenliang |
| author_facet | Tang, Yolo Y. Guo, Junjia Hua, Hang Liang, Susan Feng, Mingqian Li, Xinyang Mao, Rui Huang, Chao Bi, Jing Zhang, Zeliang Fazli, Pooyan Xu, Chenliang |
| contents | The advancement of Multimodal Large Language Models (MLLMs) has enabled significant progress in multimodal understanding, expanding their capacity to analyze video content. However, existing evaluation benchmarks for MLLMs primarily focus on abstract video comprehension, lacking a detailed assessment of their ability to understand video compositions, the nuanced interpretation of how visual elements combine and interact within highly compiled video contexts. We introduce VidComposition, a new benchmark specifically designed to evaluate the video composition understanding capabilities of MLLMs using carefully curated compiled videos and cinematic-level annotations. VidComposition includes 982 videos with 1706 multiple-choice questions, covering various compositional aspects such as camera movement, angle, shot size, narrative structure, character actions and emotions, etc. Our comprehensive evaluation of 33 open-source and proprietary MLLMs reveals a significant performance gap between human and model capabilities. This highlights the limitations of current MLLMs in understanding complex, compiled video compositions and offers insights into areas for further improvement. The leaderboard and evaluation code are available at https://yunlong10.github.io/VidComposition/ |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_10979 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | VidComposition: Can MLLMs Analyze Compositions in Compiled Videos? Tang, Yolo Y. Guo, Junjia Hua, Hang Liang, Susan Feng, Mingqian Li, Xinyang Mao, Rui Huang, Chao Bi, Jing Zhang, Zeliang Fazli, Pooyan Xu, Chenliang Computer Vision and Pattern Recognition Artificial Intelligence The advancement of Multimodal Large Language Models (MLLMs) has enabled significant progress in multimodal understanding, expanding their capacity to analyze video content. However, existing evaluation benchmarks for MLLMs primarily focus on abstract video comprehension, lacking a detailed assessment of their ability to understand video compositions, the nuanced interpretation of how visual elements combine and interact within highly compiled video contexts. We introduce VidComposition, a new benchmark specifically designed to evaluate the video composition understanding capabilities of MLLMs using carefully curated compiled videos and cinematic-level annotations. VidComposition includes 982 videos with 1706 multiple-choice questions, covering various compositional aspects such as camera movement, angle, shot size, narrative structure, character actions and emotions, etc. Our comprehensive evaluation of 33 open-source and proprietary MLLMs reveals a significant performance gap between human and model capabilities. This highlights the limitations of current MLLMs in understanding complex, compiled video compositions and offers insights into areas for further improvement. The leaderboard and evaluation code are available at https://yunlong10.github.io/VidComposition/ |
| title | VidComposition: Can MLLMs Analyze Compositions in Compiled Videos? |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence |
| url | https://arxiv.org/abs/2411.10979 |