PowerAttention: Exponentially Scaling of Receptive Fields for Effective Sparse Attention
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866917946508443648 |
|---|---|
| author | Chen, Lida Xu, Dong An, Chenxin Wang, Xintao Zhang, Yikai Chen, Jiangjie Liang, Zujie Wei, Feng Liang, Jiaqing Xiao, Yanghua Wang, Wei |
| author_facet | Chen, Lida Xu, Dong An, Chenxin Wang, Xintao Zhang, Yikai Chen, Jiangjie Liang, Zujie Wei, Feng Liang, Jiaqing Xiao, Yanghua Wang, Wei |
| contents | Large Language Models (LLMs) face efficiency bottlenecks due to the quadratic complexity of the attention mechanism when processing long contexts. Sparse attention methods offer a promising solution, but existing approaches often suffer from incomplete effective context and/or require complex implementation of pipeline. We present a comprehensive analysis of sparse attention for autoregressive LLMs from the respective of receptive field, recognize the suboptimal nature of existing methods for expanding the receptive field, and introduce PowerAttention, a novel sparse attention design that facilitates effective and complete context extension through the theoretical analysis. PowerAttention achieves exponential receptive field growth in $d$-layer LLMs, allowing each output token to attend to $2^d$ tokens, ensuring completeness and continuity of the receptive field. Experiments demonstrate that PowerAttention outperforms existing static sparse attention methods by $5\sim 40\%$, especially on tasks demanding long-range dependencies like Passkey Retrieval and RULER, while maintaining a comparable time complexity to sliding window attention. Efficiency evaluations further highlight PowerAttention's superior speedup in both prefilling and decoding phases compared with dynamic sparse attentions and full attention ($3.0\times$ faster on 128K context), making it a highly effective and user-friendly solution for processing long sequences in LLMs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_03588 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | PowerAttention: Exponentially Scaling of Receptive Fields for Effective Sparse Attention Chen, Lida Xu, Dong An, Chenxin Wang, Xintao Zhang, Yikai Chen, Jiangjie Liang, Zujie Wei, Feng Liang, Jiaqing Xiao, Yanghua Wang, Wei Computation and Language Machine Learning Large Language Models (LLMs) face efficiency bottlenecks due to the quadratic complexity of the attention mechanism when processing long contexts. Sparse attention methods offer a promising solution, but existing approaches often suffer from incomplete effective context and/or require complex implementation of pipeline. We present a comprehensive analysis of sparse attention for autoregressive LLMs from the respective of receptive field, recognize the suboptimal nature of existing methods for expanding the receptive field, and introduce PowerAttention, a novel sparse attention design that facilitates effective and complete context extension through the theoretical analysis. PowerAttention achieves exponential receptive field growth in $d$-layer LLMs, allowing each output token to attend to $2^d$ tokens, ensuring completeness and continuity of the receptive field. Experiments demonstrate that PowerAttention outperforms existing static sparse attention methods by $5\sim 40\%$, especially on tasks demanding long-range dependencies like Passkey Retrieval and RULER, while maintaining a comparable time complexity to sliding window attention. Efficiency evaluations further highlight PowerAttention's superior speedup in both prefilling and decoding phases compared with dynamic sparse attentions and full attention ($3.0\times$ faster on 128K context), making it a highly effective and user-friendly solution for processing long sequences in LLMs. |
| title | PowerAttention: Exponentially Scaling of Receptive Fields for Effective Sparse Attention |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2503.03588 |