Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2505.19155 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916021558837248 |
|---|---|
| author | Zhang, Xuan Du, Cunxiao Yu, Sicheng Wu, Jiawei Zhang, Fengzhuo Gao, Wei Liu, Qian |
| author_facet | Zhang, Xuan Du, Cunxiao Yu, Sicheng Wu, Jiawei Zhang, Fengzhuo Gao, Wei Liu, Qian |
| contents | Due to the auto-regressive nature of current video large language models (Video-LLMs), the inference latency increases as the input sequence length grows, posing challenges for the efficient processing of video sequences that are usually very long. We observe that during decoding, the attention scores of most tokens in Video-LLMs tend to be sparse and concentrated, with only certain tokens requiring comprehensive full attention. Based on this insight, we introduce Sparse-to-Dense (StD), a novel decoding strategy that integrates two distinct modules: one leveraging sparse top-K attention and the other employing dense full attention. These modules collaborate to accelerate Video-LLMs without loss. The fast (sparse) model speculatively decodes multiple tokens, while the slow (dense) model verifies them in parallel. StD is a tuning-free, plug-and-play solution that achieves up to a 1.94$\times$ walltime speedup in video processing. It maintains model performance while enabling a seamless transition from a standard Video-LLM to a sparse Video-LLM with minimal code modifications. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_19155 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Sparse-to-Dense: A Free Lunch for Lossless Acceleration of Video Understanding in LLMs Zhang, Xuan Du, Cunxiao Yu, Sicheng Wu, Jiawei Zhang, Fengzhuo Gao, Wei Liu, Qian Computer Vision and Pattern Recognition Computation and Language Due to the auto-regressive nature of current video large language models (Video-LLMs), the inference latency increases as the input sequence length grows, posing challenges for the efficient processing of video sequences that are usually very long. We observe that during decoding, the attention scores of most tokens in Video-LLMs tend to be sparse and concentrated, with only certain tokens requiring comprehensive full attention. Based on this insight, we introduce Sparse-to-Dense (StD), a novel decoding strategy that integrates two distinct modules: one leveraging sparse top-K attention and the other employing dense full attention. These modules collaborate to accelerate Video-LLMs without loss. The fast (sparse) model speculatively decodes multiple tokens, while the slow (dense) model verifies them in parallel. StD is a tuning-free, plug-and-play solution that achieves up to a 1.94$\times$ walltime speedup in video processing. It maintains model performance while enabling a seamless transition from a standard Video-LLM to a sparse Video-LLM with minimal code modifications. |
| title | Sparse-to-Dense: A Free Lunch for Lossless Acceleration of Video Understanding in LLMs |
| topic | Computer Vision and Pattern Recognition Computation and Language |
| url | https://arxiv.org/abs/2505.19155 |