VideoChat-Flash: Hierarchical Compression for Long-Context Video Modeling
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866915384726126592 |
|---|---|
| author | Li, Xinhao Wang, Yi Yu, Jiashuo Zeng, Xiangyu Zhu, Yuhan Huang, Haian Gao, Jianfei Li, Kunchang He, Yinan Wang, Chenting Qiao, Yu Wang, Yali Wang, Limin |
| author_facet | Li, Xinhao Wang, Yi Yu, Jiashuo Zeng, Xiangyu Zhu, Yuhan Huang, Haian Gao, Jianfei Li, Kunchang He, Yinan Wang, Chenting Qiao, Yu Wang, Yali Wang, Limin |
| contents | Long-context video modeling is critical for multimodal large language models (MLLMs), enabling them to process movies, online video streams, and so on. Despite its advances, handling long videos remains challenging due to the difficulty in efficiently understanding the extremely long video context. This paper aims to address this issue from aspects of model architecture, training data, training strategy and evaluation benchmark. First, we propose a novel Hierarchical video token Compression (HiCo) method, which leverages visual redundancy in long videos to compress long video context from Clip-level to Video-level, reducing the computation significantly while preserving essential details, achieving an extreme compression ratio of approximately 1/50 with almost no performance loss. Second, we introduce a multi-stage short-to-long learning scheme, a large-scale dataset of real-world long videos named LongVid, and a challenging ``Multi-Hop Needle-In-A-Video-Haystack'' benchmark. Finally, we build a powerful video MLLM named VideoChat-Flash, which shows a leading performance on both mainstream long and short video benchmarks at the 2B and 7B model scale. It first gets 99.1% accuracy over 10,000 frames in NIAH among open-source models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_00574 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | VideoChat-Flash: Hierarchical Compression for Long-Context Video Modeling Li, Xinhao Wang, Yi Yu, Jiashuo Zeng, Xiangyu Zhu, Yuhan Huang, Haian Gao, Jianfei Li, Kunchang He, Yinan Wang, Chenting Qiao, Yu Wang, Yali Wang, Limin Computer Vision and Pattern Recognition Machine Learning Long-context video modeling is critical for multimodal large language models (MLLMs), enabling them to process movies, online video streams, and so on. Despite its advances, handling long videos remains challenging due to the difficulty in efficiently understanding the extremely long video context. This paper aims to address this issue from aspects of model architecture, training data, training strategy and evaluation benchmark. First, we propose a novel Hierarchical video token Compression (HiCo) method, which leverages visual redundancy in long videos to compress long video context from Clip-level to Video-level, reducing the computation significantly while preserving essential details, achieving an extreme compression ratio of approximately 1/50 with almost no performance loss. Second, we introduce a multi-stage short-to-long learning scheme, a large-scale dataset of real-world long videos named LongVid, and a challenging ``Multi-Hop Needle-In-A-Video-Haystack'' benchmark. Finally, we build a powerful video MLLM named VideoChat-Flash, which shows a leading performance on both mainstream long and short video benchmarks at the 2B and 7B model scale. It first gets 99.1% accuracy over 10,000 frames in NIAH among open-source models. |
| title | VideoChat-Flash: Hierarchical Compression for Long-Context Video Modeling |
| topic | Computer Vision and Pattern Recognition Machine Learning |
| url | https://arxiv.org/abs/2501.00574 |