Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2603.00198 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912931844718592 |
|---|---|
| author | Jiang, Jindong Deshmukh, Amala Sanjay Chumachenko, Kateryna Sapra, Karan Yu, Zhiding Liu, Guilin Tao, Andrew Molchanov, Pavlo Kautz, Jan Byeon, Wonmin |
| author_facet | Jiang, Jindong Deshmukh, Amala Sanjay Chumachenko, Kateryna Sapra, Karan Yu, Zhiding Liu, Guilin Tao, Andrew Molchanov, Pavlo Kautz, Jan Byeon, Wonmin |
| contents | Token reduction is an effective way to accelerate long-video vision-language models (VLMs), but most existing methods are designed for dense Transformers and do not directly account for hybrid architectures that interleave attention with linear-time state-space blocks (e.g., Mamba). We study query-conditioned token reduction for hybrid video VLMs and analyze reduction behavior through two properties: layerwise sparsity (how many tokens capture query-relevant information) and importance stability (whether token-importance rankings persist across depth). Although token importance is sparse within each layer, the set of important tokens changes across layers, so aggressive early pruning is unreliable. Motivated by this, we propose a low-to-high progressive reduction schedule and a unified language-aware scoring mechanism for both attention and Mamba blocks (using an implicit-attention proxy for Mamba), enabling all-layer token reduction in hybrids. Under an aggressive compression setting (retaining 25% of visual tokens), our approach delivers substantial prefilling speedups (3.8--4.2x) with near-baseline accuracy at test time, and light finetuning under reduction further improves performance on long-context video benchmarks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_00198 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Stateful Token Reduction for Long-Video Hybrid VLMs Jiang, Jindong Deshmukh, Amala Sanjay Chumachenko, Kateryna Sapra, Karan Yu, Zhiding Liu, Guilin Tao, Andrew Molchanov, Pavlo Kautz, Jan Byeon, Wonmin Computer Vision and Pattern Recognition Artificial Intelligence Token reduction is an effective way to accelerate long-video vision-language models (VLMs), but most existing methods are designed for dense Transformers and do not directly account for hybrid architectures that interleave attention with linear-time state-space blocks (e.g., Mamba). We study query-conditioned token reduction for hybrid video VLMs and analyze reduction behavior through two properties: layerwise sparsity (how many tokens capture query-relevant information) and importance stability (whether token-importance rankings persist across depth). Although token importance is sparse within each layer, the set of important tokens changes across layers, so aggressive early pruning is unreliable. Motivated by this, we propose a low-to-high progressive reduction schedule and a unified language-aware scoring mechanism for both attention and Mamba blocks (using an implicit-attention proxy for Mamba), enabling all-layer token reduction in hybrids. Under an aggressive compression setting (retaining 25% of visual tokens), our approach delivers substantial prefilling speedups (3.8--4.2x) with near-baseline accuracy at test time, and light finetuning under reduction further improves performance on long-context video benchmarks. |
| title | Stateful Token Reduction for Long-Video Hybrid VLMs |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence |
| url | https://arxiv.org/abs/2603.00198 |