Small Vision-Language Models are Smart Compressors for Long Video Understanding
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917395488047104 |
|---|---|
| author | Fei, Junjie Chen, Jun Liu, Zechun Xiong, Yunyang Zhou, Chong Wen, Wei Han, Junlin Zhuge, Mingchen Suri, Saksham Qian, Qi Liu, Shuming Wu, Lemeng Krishnamoorthi, Raghuraman Chandra, Vikas Elhoseiny, Mohamed Zhu, Chenchen |
| author_facet | Fei, Junjie Chen, Jun Liu, Zechun Xiong, Yunyang Zhou, Chong Wen, Wei Han, Junlin Zhuge, Mingchen Suri, Saksham Qian, Qi Liu, Shuming Wu, Lemeng Krishnamoorthi, Raghuraman Chandra, Vikas Elhoseiny, Mohamed Zhu, Chenchen |
| contents | Adapting Multimodal Large Language Models (MLLMs) for hour-long videos is bottlenecked by context limits. Dense visual streams saturate token budgets and exacerbate the lost-in-the-middle phenomenon. Existing heuristics, like sparse sampling or uniform pooling, blindly sacrifice fidelity by discarding decisive moments and wasting bandwidth on irrelevant backgrounds. We propose Tempo, an efficient query-aware framework compressing long videos for downstream understanding. Tempo leverages a Small Vision-Language Model (SVLM) as a local temporal compressor, casting token reduction as an early cross-modal distillation process to generate compact, intent-aligned representations in a single forward pass. To enforce strict budgets without breaking causality, we introduce Adaptive Token Allocation (ATA). Exploiting the SVLM's zero-shot relevance prior and semantic front-loading, ATA acts as a training-free $O(1)$ dynamic router. It allocates dense bandwidth to query-critical segments while compressing redundancies into minimal temporal anchors to maintain the global storyline. Extensive experiments show our 6B architecture achieves state-of-the-art performance with aggressive dynamic compression (0.5-16 tokens/frame). On the extreme-long LVBench (4101s), Tempo scores 52.3 under a strict 8K visual budget, outperforming GPT-4o and Gemini 1.5 Pro. Scaling to 2048 frames reaches 53.7. Crucially, Tempo compresses hour-long videos substantially below theoretical limits, proving true long-form video understanding relies on intent-driven efficiency rather than greedily padded context windows. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_08120 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Small Vision-Language Models are Smart Compressors for Long Video Understanding Fei, Junjie Chen, Jun Liu, Zechun Xiong, Yunyang Zhou, Chong Wen, Wei Han, Junlin Zhuge, Mingchen Suri, Saksham Qian, Qi Liu, Shuming Wu, Lemeng Krishnamoorthi, Raghuraman Chandra, Vikas Elhoseiny, Mohamed Zhu, Chenchen Computer Vision and Pattern Recognition Artificial Intelligence Computation and Language Machine Learning Adapting Multimodal Large Language Models (MLLMs) for hour-long videos is bottlenecked by context limits. Dense visual streams saturate token budgets and exacerbate the lost-in-the-middle phenomenon. Existing heuristics, like sparse sampling or uniform pooling, blindly sacrifice fidelity by discarding decisive moments and wasting bandwidth on irrelevant backgrounds. We propose Tempo, an efficient query-aware framework compressing long videos for downstream understanding. Tempo leverages a Small Vision-Language Model (SVLM) as a local temporal compressor, casting token reduction as an early cross-modal distillation process to generate compact, intent-aligned representations in a single forward pass. To enforce strict budgets without breaking causality, we introduce Adaptive Token Allocation (ATA). Exploiting the SVLM's zero-shot relevance prior and semantic front-loading, ATA acts as a training-free $O(1)$ dynamic router. It allocates dense bandwidth to query-critical segments while compressing redundancies into minimal temporal anchors to maintain the global storyline. Extensive experiments show our 6B architecture achieves state-of-the-art performance with aggressive dynamic compression (0.5-16 tokens/frame). On the extreme-long LVBench (4101s), Tempo scores 52.3 under a strict 8K visual budget, outperforming GPT-4o and Gemini 1.5 Pro. Scaling to 2048 frames reaches 53.7. Crucially, Tempo compresses hour-long videos substantially below theoretical limits, proving true long-form video understanding relies on intent-driven efficiency rather than greedily padded context windows. |
| title | Small Vision-Language Models are Smart Compressors for Long Video Understanding |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2604.08120 |