StreamingAssistant: Efficient Visual Token Pruning for Accelerating Online Video Understanding
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866908709249089536 |
|---|---|
| author | Jin, Xinqi Yu, Hanxun Yu, Bohan Liu, Kebin Liu, Jian Tao, Keda Pei, Yixuan Wang, Huan Dang, Fan Liu, Jiangchuan Wang, Weiqiang |
| author_facet | Jin, Xinqi Yu, Hanxun Yu, Bohan Liu, Kebin Liu, Jian Tao, Keda Pei, Yixuan Wang, Huan Dang, Fan Liu, Jiangchuan Wang, Weiqiang |
| contents | Online video understanding is essential for applications like public surveillance and AI glasses. However, applying Multimodal Large Language Models (MLLMs) to this domain is challenging due to the large number of video frames, resulting in high GPU memory usage and computational latency. To address these challenges, we propose token pruning as a means to reduce context length while retaining critical information. Specifically, we introduce a novel redundancy metric, Maximum Similarity to Spatially Adjacent Video Tokens (MSSAVT), which accounts for both token similarity and spatial position. To mitigate the bidirectional dependency between pruning and redundancy, we further design a masked pruning strategy that ensures only mutually unadjacent tokens are pruned. We also integrate an existing temporal redundancy-based pruning method to eliminate temporal redundancy of the video modality. Experimental results on multiple online and offline video understanding benchmarks demonstrate that our method significantly improves the accuracy (i.e., by 4\% at most) while incurring a negligible pruning latency (i.e., less than 1ms). Our full implementation will be made publicly available. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_12560 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | StreamingAssistant: Efficient Visual Token Pruning for Accelerating Online Video Understanding Jin, Xinqi Yu, Hanxun Yu, Bohan Liu, Kebin Liu, Jian Tao, Keda Pei, Yixuan Wang, Huan Dang, Fan Liu, Jiangchuan Wang, Weiqiang Computer Vision and Pattern Recognition Artificial Intelligence Online video understanding is essential for applications like public surveillance and AI glasses. However, applying Multimodal Large Language Models (MLLMs) to this domain is challenging due to the large number of video frames, resulting in high GPU memory usage and computational latency. To address these challenges, we propose token pruning as a means to reduce context length while retaining critical information. Specifically, we introduce a novel redundancy metric, Maximum Similarity to Spatially Adjacent Video Tokens (MSSAVT), which accounts for both token similarity and spatial position. To mitigate the bidirectional dependency between pruning and redundancy, we further design a masked pruning strategy that ensures only mutually unadjacent tokens are pruned. We also integrate an existing temporal redundancy-based pruning method to eliminate temporal redundancy of the video modality. Experimental results on multiple online and offline video understanding benchmarks demonstrate that our method significantly improves the accuracy (i.e., by 4\% at most) while incurring a negligible pruning latency (i.e., less than 1ms). Our full implementation will be made publicly available. |
| title | StreamingAssistant: Efficient Visual Token Pruning for Accelerating Online Video Understanding |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence |
| url | https://arxiv.org/abs/2512.12560 |