Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2409.18938 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929611202363392 |
|---|---|
| author | Zou, Heqing Luo, Tianze Xie, Guiyang Victor Zhang Lv, Fengmao Wang, Guangcong Chen, Junyang Wang, Zhuochen Zhang, Hansheng Zhang, Huaijian |
| author_facet | Zou, Heqing Luo, Tianze Xie, Guiyang Victor Zhang Lv, Fengmao Wang, Guangcong Chen, Junyang Wang, Zhuochen Zhang, Hansheng Zhang, Huaijian |
| contents | The integration of Large Language Models (LLMs) with visual encoders has recently shown promising performance in visual understanding tasks, leveraging their inherent capability to comprehend and generate human-like text for visual reasoning. Given the diverse nature of visual data, MultiModal Large Language Models (MM-LLMs) exhibit variations in model designing and training for understanding images, short videos, and long videos. Our paper focuses on the substantial differences and unique challenges posed by long video understanding compared to static image and short video understanding. Unlike static images, short videos encompass sequential frames with both spatial and within-event temporal information, while long videos consist of multiple events with between-event and long-term temporal information. In this survey, we aim to trace and summarize the advancements of MM-LLMs from image understanding to long video understanding. We review the differences among various visual understanding tasks and highlight the challenges in long video understanding, including more fine-grained spatiotemporal details, dynamic events, and long-term dependencies. We then provide a detailed summary of the advancements in MM-LLMs in terms of model design and training methodologies for understanding long videos. Finally, we compare the performance of existing MM-LLMs on video understanding benchmarks of various lengths and discuss potential future directions for MM-LLMs in long video understanding. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_18938 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | From Seconds to Hours: Reviewing MultiModal Large Language Models on Comprehensive Long Video Understanding Zou, Heqing Luo, Tianze Xie, Guiyang Victor Zhang Lv, Fengmao Wang, Guangcong Chen, Junyang Wang, Zhuochen Zhang, Hansheng Zhang, Huaijian Computer Vision and Pattern Recognition Artificial Intelligence The integration of Large Language Models (LLMs) with visual encoders has recently shown promising performance in visual understanding tasks, leveraging their inherent capability to comprehend and generate human-like text for visual reasoning. Given the diverse nature of visual data, MultiModal Large Language Models (MM-LLMs) exhibit variations in model designing and training for understanding images, short videos, and long videos. Our paper focuses on the substantial differences and unique challenges posed by long video understanding compared to static image and short video understanding. Unlike static images, short videos encompass sequential frames with both spatial and within-event temporal information, while long videos consist of multiple events with between-event and long-term temporal information. In this survey, we aim to trace and summarize the advancements of MM-LLMs from image understanding to long video understanding. We review the differences among various visual understanding tasks and highlight the challenges in long video understanding, including more fine-grained spatiotemporal details, dynamic events, and long-term dependencies. We then provide a detailed summary of the advancements in MM-LLMs in terms of model design and training methodologies for understanding long videos. Finally, we compare the performance of existing MM-LLMs on video understanding benchmarks of various lengths and discuss potential future directions for MM-LLMs in long video understanding. |
| title | From Seconds to Hours: Reviewing MultiModal Large Language Models on Comprehensive Long Video Understanding |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence |
| url | https://arxiv.org/abs/2409.18938 |