LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understanding
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913560806817792 |
|---|---|
| author | Shen, Xiaoqian Xiong, Yunyang Zhao, Changsheng Wu, Lemeng Chen, Jun Zhu, Chenchen Liu, Zechun Xiao, Fanyi Varadarajan, Balakrishnan Bordes, Florian Liu, Zhuang Xu, Hu Kim, Hyunwoo J. Soran, Bilge Krishnamoorthi, Raghuraman Elhoseiny, Mohamed Chandra, Vikas |
| author_facet | Shen, Xiaoqian Xiong, Yunyang Zhao, Changsheng Wu, Lemeng Chen, Jun Zhu, Chenchen Liu, Zechun Xiao, Fanyi Varadarajan, Balakrishnan Bordes, Florian Liu, Zhuang Xu, Hu Kim, Hyunwoo J. Soran, Bilge Krishnamoorthi, Raghuraman Elhoseiny, Mohamed Chandra, Vikas |
| contents | Multimodal Large Language Models (MLLMs) have shown promising progress in understanding and analyzing video content. However, processing long videos remains a significant challenge constrained by LLM's context size. To address this limitation, we propose LongVU, a spatiotemporal adaptive compression mechanism thats reduces the number of video tokens while preserving visual details of long videos. Our idea is based on leveraging cross-modal query and inter-frame dependencies to adaptively reduce temporal and spatial redundancy in videos. Specifically, we leverage DINOv2 features to remove redundant frames that exhibit high similarity. Then we utilize text-guided cross-modal query for selective frame feature reduction. Further, we perform spatial token reduction across frames based on their temporal dependencies. Our adaptive compression strategy effectively processes a large number of frames with little visual information loss within given context length. Our LongVU consistently surpass existing methods across a variety of video understanding benchmarks, especially on hour-long video understanding tasks such as VideoMME and MLVU. Given a light-weight LLM, our LongVU also scales effectively into a smaller size with state-of-the-art video understanding performance. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_17434 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understanding Shen, Xiaoqian Xiong, Yunyang Zhao, Changsheng Wu, Lemeng Chen, Jun Zhu, Chenchen Liu, Zechun Xiao, Fanyi Varadarajan, Balakrishnan Bordes, Florian Liu, Zhuang Xu, Hu Kim, Hyunwoo J. Soran, Bilge Krishnamoorthi, Raghuraman Elhoseiny, Mohamed Chandra, Vikas Computer Vision and Pattern Recognition Multimodal Large Language Models (MLLMs) have shown promising progress in understanding and analyzing video content. However, processing long videos remains a significant challenge constrained by LLM's context size. To address this limitation, we propose LongVU, a spatiotemporal adaptive compression mechanism thats reduces the number of video tokens while preserving visual details of long videos. Our idea is based on leveraging cross-modal query and inter-frame dependencies to adaptively reduce temporal and spatial redundancy in videos. Specifically, we leverage DINOv2 features to remove redundant frames that exhibit high similarity. Then we utilize text-guided cross-modal query for selective frame feature reduction. Further, we perform spatial token reduction across frames based on their temporal dependencies. Our adaptive compression strategy effectively processes a large number of frames with little visual information loss within given context length. Our LongVU consistently surpass existing methods across a variety of video understanding benchmarks, especially on hour-long video understanding tasks such as VideoMME and MLVU. Given a light-weight LLM, our LongVU also scales effectively into a smaller size with state-of-the-art video understanding performance. |
| title | LongVU: Spatiotemporal Adaptive Compression for Long Video-Language Understanding |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2410.17434 |