Twilight: Adaptive Attention Sparsity with Hierarchical Top-$p$ Pruning
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866909886361632768 |
|---|---|
| author | Lin, Chaofan Tang, Jiaming Yang, Shuo Wang, Hanshuo Tang, Tian Tian, Boyu Stoica, Ion Han, Song Gao, Mingyu |
| author_facet | Lin, Chaofan Tang, Jiaming Yang, Shuo Wang, Hanshuo Tang, Tian Tian, Boyu Stoica, Ion Han, Song Gao, Mingyu |
| contents | Leveraging attention sparsity to accelerate long-context large language models (LLMs) has been a hot research topic. However, current algorithms such as sparse attention or key-value (KV) cache compression tend to use a fixed budget, which presents a significant challenge during deployment because it fails to account for the dynamic nature of real-world scenarios, where the optimal balance between accuracy and efficiency can vary greatly. In this paper, we find that borrowing top-$p$ sampling (nucleus sampling) to sparse attention can surprisingly achieve adaptive budgeting. Based on this, we propose Twilight, a framework to bring adaptive sparsity to any existing sparse attention algorithm without sacrificing their accuracy. Empirical results show that Twilight can adaptively prune at most 98% of redundant tokens, leading to $15.4\times$ acceleration in self-attention operations and $3.9\times$ acceleration in end-to-end per token latency in long context LLM decoding. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_02770 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Twilight: Adaptive Attention Sparsity with Hierarchical Top-$p$ Pruning Lin, Chaofan Tang, Jiaming Yang, Shuo Wang, Hanshuo Tang, Tian Tian, Boyu Stoica, Ion Han, Song Gao, Mingyu Machine Learning Computation and Language Leveraging attention sparsity to accelerate long-context large language models (LLMs) has been a hot research topic. However, current algorithms such as sparse attention or key-value (KV) cache compression tend to use a fixed budget, which presents a significant challenge during deployment because it fails to account for the dynamic nature of real-world scenarios, where the optimal balance between accuracy and efficiency can vary greatly. In this paper, we find that borrowing top-$p$ sampling (nucleus sampling) to sparse attention can surprisingly achieve adaptive budgeting. Based on this, we propose Twilight, a framework to bring adaptive sparsity to any existing sparse attention algorithm without sacrificing their accuracy. Empirical results show that Twilight can adaptively prune at most 98% of redundant tokens, leading to $15.4\times$ acceleration in self-attention operations and $3.9\times$ acceleration in end-to-end per token latency in long context LLM decoding. |
| title | Twilight: Adaptive Attention Sparsity with Hierarchical Top-$p$ Pruning |
| topic | Machine Learning Computation and Language |
| url | https://arxiv.org/abs/2502.02770 |