BLASST: Dynamic BLocked Attention Sparsity via Softmax Thresholding
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908996549476352 |
|---|---|
| author | Yuan, Jiayi Shinn, Cameron Xu, Kai Cui, Jingze Klimiashvili, George Xiao, Guangxuan Zheng, Perkz Li, Bo Zhou, Yuxin Ye, Zhouhai You, Weijie Zheng, Tian Brown, Dominic Wang, Pengbo Hoehnerbach, Markus Cai, Richard Demouth, Julien Owens, John D. Hu, Xia Han, Song Liu, Timmy Mao, Huizi |
| author_facet | Yuan, Jiayi Shinn, Cameron Xu, Kai Cui, Jingze Klimiashvili, George Xiao, Guangxuan Zheng, Perkz Li, Bo Zhou, Yuxin Ye, Zhouhai You, Weijie Zheng, Tian Brown, Dominic Wang, Pengbo Hoehnerbach, Markus Cai, Richard Demouth, Julien Owens, John D. Hu, Xia Han, Song Liu, Timmy Mao, Huizi |
| contents | The growing demand for long-context inference capabilities in Large Language Models (LLMs) has intensified the computational and memory bottlenecks inherent to the self-attention mechanism. To address this challenge, we introduce BLASST, a drop-in, dynamic sparse attention mechanism that accelerates inference by using only a fixed scalar threshold to skip attention blocks. Our method targets practical inference deployment by removing the barriers to adoption present in existing works. As such, BLASST eliminates training requirements, avoids expensive pre-computation passes, accelerates both prefill and decode across all major attention variants (MHA, GQA, MQA, and MLA), provides optimized support for modern hardware, and easily integrates into existing frameworks. This is achieved by reusing online softmax statistics to identify negligible attention scores, skipping softmax, value block loads, and the subsequent matrix multiplication. We demonstrate the BLASST algorithm by delivering optimized kernels with negligible latency overhead. Our automated threshold calibration procedure reveals a simple inverse relationship between optimal threshold and context length, meaning we require only a single threshold each for prefill and decode per model. Preserving benchmark accuracy, we demonstrate a 1.52x speedup for prefill at 71.9% sparsity and a 1.48x speedup for decode at 73.2% sparsity on modern GPUs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_12087 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | BLASST: Dynamic BLocked Attention Sparsity via Softmax Thresholding Yuan, Jiayi Shinn, Cameron Xu, Kai Cui, Jingze Klimiashvili, George Xiao, Guangxuan Zheng, Perkz Li, Bo Zhou, Yuxin Ye, Zhouhai You, Weijie Zheng, Tian Brown, Dominic Wang, Pengbo Hoehnerbach, Markus Cai, Richard Demouth, Julien Owens, John D. Hu, Xia Han, Song Liu, Timmy Mao, Huizi Computation and Language The growing demand for long-context inference capabilities in Large Language Models (LLMs) has intensified the computational and memory bottlenecks inherent to the self-attention mechanism. To address this challenge, we introduce BLASST, a drop-in, dynamic sparse attention mechanism that accelerates inference by using only a fixed scalar threshold to skip attention blocks. Our method targets practical inference deployment by removing the barriers to adoption present in existing works. As such, BLASST eliminates training requirements, avoids expensive pre-computation passes, accelerates both prefill and decode across all major attention variants (MHA, GQA, MQA, and MLA), provides optimized support for modern hardware, and easily integrates into existing frameworks. This is achieved by reusing online softmax statistics to identify negligible attention scores, skipping softmax, value block loads, and the subsequent matrix multiplication. We demonstrate the BLASST algorithm by delivering optimized kernels with negligible latency overhead. Our automated threshold calibration procedure reveals a simple inverse relationship between optimal threshold and context length, meaning we require only a single threshold each for prefill and decode per model. Preserving benchmark accuracy, we demonstrate a 1.52x speedup for prefill at 71.9% sparsity and a 1.48x speedup for decode at 73.2% sparsity on modern GPUs. |
| title | BLASST: Dynamic BLocked Attention Sparsity via Softmax Thresholding |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2512.12087 |