SeerAttention-R: Sparse Attention Adaptation for Long Reasoning
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915335451443200 |
|---|---|
| author | Gao, Yizhao Guo, Shuming Cao, Shijie Xia, Yuqing Cheng, Yu Wang, Lei Ma, Lingxiao Sun, Yutao Ye, Tianzhu Dong, Li So, Hayden Kwok-Hay Hua, Yu Cao, Ting Yang, Fan Yang, Mao |
| author_facet | Gao, Yizhao Guo, Shuming Cao, Shijie Xia, Yuqing Cheng, Yu Wang, Lei Ma, Lingxiao Sun, Yutao Ye, Tianzhu Dong, Li So, Hayden Kwok-Hay Hua, Yu Cao, Ting Yang, Fan Yang, Mao |
| contents | We introduce SeerAttention-R, a sparse attention framework specifically tailored for the long decoding of reasoning models. Extended from SeerAttention, SeerAttention-R retains the design of learning attention sparsity through a self-distilled gating mechanism, while removing query pooling to accommodate auto-regressive decoding. With a lightweight plug-in gating, SeerAttention-R is flexible and can be easily integrated into existing pretrained model without modifying the original parameters. We demonstrate that SeerAttention-R, trained on just 0.4B tokens, maintains near-lossless reasoning accuracy with 4K token budget in AIME benchmark under large sparse attention block sizes (64/128). Using TileLang, we develop a highly optimized sparse decoding kernel that achieves near-theoretical speedups of up to 9x over FlashAttention-3 on H100 GPU at 90% sparsity. Code is available at: https://github.com/microsoft/SeerAttention. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_08889 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | SeerAttention-R: Sparse Attention Adaptation for Long Reasoning Gao, Yizhao Guo, Shuming Cao, Shijie Xia, Yuqing Cheng, Yu Wang, Lei Ma, Lingxiao Sun, Yutao Ye, Tianzhu Dong, Li So, Hayden Kwok-Hay Hua, Yu Cao, Ting Yang, Fan Yang, Mao Machine Learning Artificial Intelligence We introduce SeerAttention-R, a sparse attention framework specifically tailored for the long decoding of reasoning models. Extended from SeerAttention, SeerAttention-R retains the design of learning attention sparsity through a self-distilled gating mechanism, while removing query pooling to accommodate auto-regressive decoding. With a lightweight plug-in gating, SeerAttention-R is flexible and can be easily integrated into existing pretrained model without modifying the original parameters. We demonstrate that SeerAttention-R, trained on just 0.4B tokens, maintains near-lossless reasoning accuracy with 4K token budget in AIME benchmark under large sparse attention block sizes (64/128). Using TileLang, we develop a highly optimized sparse decoding kernel that achieves near-theoretical speedups of up to 9x over FlashAttention-3 on H100 GPU at 90% sparsity. Code is available at: https://github.com/microsoft/SeerAttention. |
| title | SeerAttention-R: Sparse Attention Adaptation for Long Reasoning |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2506.08889 |