OmniSparse: Training-Aware Fine-Grained Sparse Attention for Long-Video MLLMs
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866917090037858304 |
|---|---|
| author | Chen, Feng He, Yefei He, Shaoxuan He, Yuanyu Liu, Jing Lin, Lequan Liu, Akide Li, Zhaoyang Zhang, Jiyuan Sun, Zhenbang Zhuang, Bohan Wu, Qi |
| author_facet | Chen, Feng He, Yefei He, Shaoxuan He, Yuanyu Liu, Jing Lin, Lequan Liu, Akide Li, Zhaoyang Zhang, Jiyuan Sun, Zhenbang Zhuang, Bohan Wu, Qi |
| contents | Existing sparse attention methods primarily target inference-time acceleration by selecting critical tokens under predefined sparsity patterns. However, they often fail to bridge the training-inference gap and lack the capacity for fine-grained token selection across multiple dimensions such as queries, key-values (KV), and heads, leading to suboptimal performance and limited acceleration gains. In this paper, we introduce OmniSparse, a training-aware fine-grained sparse attention framework for long-video MLLMs, which operates in both training and inference with dynamic token budget allocation. Specifically, OmniSparse contains three adaptive and complementary mechanisms: (1) query selection via lazy-active classification, retaining active queries that capture broad semantic similarity while discarding most lazy ones that focus on limited local context and exhibit high functional redundancy; (2) KV selection with head-level dynamic budget allocation, where a shared budget is determined based on the flattest head and applied uniformly across all heads to ensure attention recall; and (3) KV cache slimming to reduce head-level redundancy by selectively fetching visual KV cache according to the head-level decoding query pattern. Experimental results show that OmniSparse matches the performance of full attention while achieving up to 2.7x speedup during prefill and 2.4x memory reduction during decoding. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_12201 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | OmniSparse: Training-Aware Fine-Grained Sparse Attention for Long-Video MLLMs Chen, Feng He, Yefei He, Shaoxuan He, Yuanyu Liu, Jing Lin, Lequan Liu, Akide Li, Zhaoyang Zhang, Jiyuan Sun, Zhenbang Zhuang, Bohan Wu, Qi Computer Vision and Pattern Recognition Existing sparse attention methods primarily target inference-time acceleration by selecting critical tokens under predefined sparsity patterns. However, they often fail to bridge the training-inference gap and lack the capacity for fine-grained token selection across multiple dimensions such as queries, key-values (KV), and heads, leading to suboptimal performance and limited acceleration gains. In this paper, we introduce OmniSparse, a training-aware fine-grained sparse attention framework for long-video MLLMs, which operates in both training and inference with dynamic token budget allocation. Specifically, OmniSparse contains three adaptive and complementary mechanisms: (1) query selection via lazy-active classification, retaining active queries that capture broad semantic similarity while discarding most lazy ones that focus on limited local context and exhibit high functional redundancy; (2) KV selection with head-level dynamic budget allocation, where a shared budget is determined based on the flattest head and applied uniformly across all heads to ensure attention recall; and (3) KV cache slimming to reduce head-level redundancy by selectively fetching visual KV cache according to the head-level decoding query pattern. Experimental results show that OmniSparse matches the performance of full attention while achieving up to 2.7x speedup during prefill and 2.4x memory reduction during decoding. |
| title | OmniSparse: Training-Aware Fine-Grained Sparse Attention for Long-Video MLLMs |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2511.12201 |