Top-Theta Attention: Sparsifying Transformers by Compensated Thresholding
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866909748417265664 |
|---|---|
| author | Berestizshevsky, Konstantin Andri, Renzo Cavigelli, Lukas |
| author_facet | Berestizshevsky, Konstantin Andri, Renzo Cavigelli, Lukas |
| contents | We present Top-Theta (Top-$θ$) Attention, a training-free method for sparsifying transformer attention during inference. Our key insight is that static, per-head thresholds can be calibrated to retain the desired constant number of significant elements per attention row. This approach enables content-based sparsity without retraining, and it remains robust across data domains. We further introduce compensation techniques to preserve accuracy under aggressive sparsification, establishing attention thresholding as a practical and principled alternative to top-k attention. We provide extensive evaluation on natural language processing tasks, showing that Top-$θ$ achieves 3-10x reduction in V-cache usage and up to 10x fewer attention elements during inference while degrading no more than 1% in accuracy. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_08363 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Top-Theta Attention: Sparsifying Transformers by Compensated Thresholding Berestizshevsky, Konstantin Andri, Renzo Cavigelli, Lukas Computation and Language Artificial Intelligence 68T01 I.2 We present Top-Theta (Top-$θ$) Attention, a training-free method for sparsifying transformer attention during inference. Our key insight is that static, per-head thresholds can be calibrated to retain the desired constant number of significant elements per attention row. This approach enables content-based sparsity without retraining, and it remains robust across data domains. We further introduce compensation techniques to preserve accuracy under aggressive sparsification, establishing attention thresholding as a practical and principled alternative to top-k attention. We provide extensive evaluation on natural language processing tasks, showing that Top-$θ$ achieves 3-10x reduction in V-cache usage and up to 10x fewer attention elements during inference while degrading no more than 1% in accuracy. |
| title | Top-Theta Attention: Sparsifying Transformers by Compensated Thresholding |
| topic | Computation and Language Artificial Intelligence 68T01 I.2 |
| url | https://arxiv.org/abs/2502.08363 |