Tactic: Adaptive Sparse Attention with Clustering and Distribution Fitting for Long-Context LLMs
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866909499303919616 |
|---|---|
| author | Zhu, Kan Tang, Tian Xu, Qinyu Gu, Yile Zeng, Zhichen Kadekodi, Rohan Zhao, Liangyu Li, Ang Krishnamurthy, Arvind Kasikci, Baris |
| author_facet | Zhu, Kan Tang, Tian Xu, Qinyu Gu, Yile Zeng, Zhichen Kadekodi, Rohan Zhao, Liangyu Li, Ang Krishnamurthy, Arvind Kasikci, Baris |
| contents | Long-context models are essential for many applications but face inefficiencies in loading large KV caches during decoding. Prior methods enforce fixed token budgets for sparse attention, assuming a set number of tokens can approximate full attention. However, these methods overlook variations in the importance of attention across heads, layers, and contexts. To address these limitations, we propose Tactic, a sparsity-adaptive and calibration-free sparse attention mechanism that dynamically selects tokens based on their cumulative attention scores rather than a fixed token budget. By setting a target fraction of total attention scores, Tactic ensures that token selection naturally adapts to variations in attention sparsity. To efficiently approximate this selection, Tactic leverages clustering-based sorting and distribution fitting, allowing it to accurately estimate token importance with minimal computational overhead. We show that Tactic outperforms existing sparse attention algorithms, achieving superior accuracy and up to 7.29x decode attention speedup. This improvement translates to an overall 1.58x end-to-end inference speedup, making Tactic a practical and effective solution for long-context LLM inference in accuracy-sensitive applications. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_12216 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Tactic: Adaptive Sparse Attention with Clustering and Distribution Fitting for Long-Context LLMs Zhu, Kan Tang, Tian Xu, Qinyu Gu, Yile Zeng, Zhichen Kadekodi, Rohan Zhao, Liangyu Li, Ang Krishnamurthy, Arvind Kasikci, Baris Machine Learning Artificial Intelligence Computation and Language Long-context models are essential for many applications but face inefficiencies in loading large KV caches during decoding. Prior methods enforce fixed token budgets for sparse attention, assuming a set number of tokens can approximate full attention. However, these methods overlook variations in the importance of attention across heads, layers, and contexts. To address these limitations, we propose Tactic, a sparsity-adaptive and calibration-free sparse attention mechanism that dynamically selects tokens based on their cumulative attention scores rather than a fixed token budget. By setting a target fraction of total attention scores, Tactic ensures that token selection naturally adapts to variations in attention sparsity. To efficiently approximate this selection, Tactic leverages clustering-based sorting and distribution fitting, allowing it to accurately estimate token importance with minimal computational overhead. We show that Tactic outperforms existing sparse attention algorithms, achieving superior accuracy and up to 7.29x decode attention speedup. This improvement translates to an overall 1.58x end-to-end inference speedup, making Tactic a practical and effective solution for long-context LLM inference in accuracy-sensitive applications. |
| title | Tactic: Adaptive Sparse Attention with Clustering and Distribution Fitting for Long-Context LLMs |
| topic | Machine Learning Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2502.12216 |