Multi-Granular Spatio-Temporal Token Merging for Training-Free Acceleration of Video LLMs
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908445015277568 |
|---|---|
| author | Hyun, Jeongseok Hwang, Sukjun Han, Su Ho Kim, Taeoh Lee, Inwoong Wee, Dongyoon Lee, Joon-Young Kim, Seon Joo Shim, Minho |
| author_facet | Hyun, Jeongseok Hwang, Sukjun Han, Su Ho Kim, Taeoh Lee, Inwoong Wee, Dongyoon Lee, Joon-Young Kim, Seon Joo Shim, Minho |
| contents | Video large language models (LLMs) achieve strong video understanding by leveraging a large number of spatio-temporal tokens, but suffer from quadratic computational scaling with token count. To address this, we propose a training-free spatio-temporal token merging method, named STTM. Our key insight is to exploit local spatial and temporal redundancy in video data which has been overlooked in prior work. STTM first transforms each frame into multi-granular spatial tokens using a coarse-to-fine search over a quadtree structure, then performs directed pairwise merging across the temporal dimension. This decomposed merging approach outperforms existing token reduction methods across six video QA benchmarks. Notably, STTM achieves a 2$\times$ speed-up with only a 0.5% accuracy drop under a 50% token budget, and a 3$\times$ speed-up with just a 2% drop under a 30% budget. Moreover, STTM is query-agnostic, allowing KV cache reuse across different questions for the same video. The project page is available at https://www.jshyun.me/projects/sttm. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_07990 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Multi-Granular Spatio-Temporal Token Merging for Training-Free Acceleration of Video LLMs Hyun, Jeongseok Hwang, Sukjun Han, Su Ho Kim, Taeoh Lee, Inwoong Wee, Dongyoon Lee, Joon-Young Kim, Seon Joo Shim, Minho Computer Vision and Pattern Recognition Artificial Intelligence Video large language models (LLMs) achieve strong video understanding by leveraging a large number of spatio-temporal tokens, but suffer from quadratic computational scaling with token count. To address this, we propose a training-free spatio-temporal token merging method, named STTM. Our key insight is to exploit local spatial and temporal redundancy in video data which has been overlooked in prior work. STTM first transforms each frame into multi-granular spatial tokens using a coarse-to-fine search over a quadtree structure, then performs directed pairwise merging across the temporal dimension. This decomposed merging approach outperforms existing token reduction methods across six video QA benchmarks. Notably, STTM achieves a 2$\times$ speed-up with only a 0.5% accuracy drop under a 50% token budget, and a 3$\times$ speed-up with just a 2% drop under a 30% budget. Moreover, STTM is query-agnostic, allowing KV cache reuse across different questions for the same video. The project page is available at https://www.jshyun.me/projects/sttm. |
| title | Multi-Granular Spatio-Temporal Token Merging for Training-Free Acceleration of Video LLMs |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence |
| url | https://arxiv.org/abs/2507.07990 |