Rectified Sparse Attention
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915326926520320 |
|---|---|
| author | Sun, Yutao Ye, Tianzhu Dong, Li Xia, Yuqing Chen, Jian Gao, Yizhao Cao, Shijie Wang, Jianyong Wei, Furu |
| author_facet | Sun, Yutao Ye, Tianzhu Dong, Li Xia, Yuqing Chen, Jian Gao, Yizhao Cao, Shijie Wang, Jianyong Wei, Furu |
| contents | Efficient long-sequence generation is a critical challenge for Large Language Models. While recent sparse decoding methods improve efficiency, they suffer from KV cache misalignment, where approximation errors accumulate and degrade generation quality. In this work, we propose Rectified Sparse Attention (ReSA), a simple yet effective method that combines block-sparse attention with periodic dense rectification. By refreshing the KV cache at fixed intervals using a dense forward pass, ReSA bounds error accumulation and preserves alignment with the pretraining distribution. Experiments across math reasoning, language modeling, and retrieval tasks demonstrate that ReSA achieves near-lossless generation quality with significantly improved efficiency. Notably, ReSA delivers up to 2.42$\times$ end-to-end speedup under decoding at 256K sequence length, making it a practical solution for scalable long-context inference. Code is available at https://aka.ms/ReSA-LM. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_04108 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Rectified Sparse Attention Sun, Yutao Ye, Tianzhu Dong, Li Xia, Yuqing Chen, Jian Gao, Yizhao Cao, Shijie Wang, Jianyong Wei, Furu Computation and Language Efficient long-sequence generation is a critical challenge for Large Language Models. While recent sparse decoding methods improve efficiency, they suffer from KV cache misalignment, where approximation errors accumulate and degrade generation quality. In this work, we propose Rectified Sparse Attention (ReSA), a simple yet effective method that combines block-sparse attention with periodic dense rectification. By refreshing the KV cache at fixed intervals using a dense forward pass, ReSA bounds error accumulation and preserves alignment with the pretraining distribution. Experiments across math reasoning, language modeling, and retrieval tasks demonstrate that ReSA achieves near-lossless generation quality with significantly improved efficiency. Notably, ReSA delivers up to 2.42$\times$ end-to-end speedup under decoding at 256K sequence length, making it a practical solution for scalable long-context inference. Code is available at https://aka.ms/ReSA-LM. |
| title | Rectified Sparse Attention |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2506.04108 |