SSA: Sparse Sparse Attention by Aligning Full and Sparse Attention Outputs in Feature Space
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866917234101714944 |
|---|---|
| author | Shen, Zhenyi Lu, Junru Gui, Lin Li, Jiazheng He, Yulan Yin, Di Sun, Xing |
| author_facet | Shen, Zhenyi Lu, Junru Gui, Lin Li, Jiazheng He, Yulan Yin, Di Sun, Xing |
| contents | Sparse attention reduces the quadratic complexity of full self-attention but faces two challenges: (1) an attention gap, where applying sparse attention to full-attention-trained models causes performance degradation due to train-inference distribution mismatch, and (2) a capability gap, where models trained purely with sparse attention lack complete gradient flow, preventing them from matching full-attention performance. We propose SSA (Sparse Sparse Attention), a training framework that integrates both sparse and full attention with bidirectional attention-output alignment. We prove that the approximation error scales linearly with the attention mass dropped under sparse attention, and show that SSA's alignment objective substantially reduces this quantity compared to baselines. Experiments demonstrate that SSA achieves state-of-the-art performance under both inference modes, adapts smoothly to varying sparsity budgets, and demonstrates superior long-context capabilities. The code is available at https://github.com/zhenyi4/ssa. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_20102 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | SSA: Sparse Sparse Attention by Aligning Full and Sparse Attention Outputs in Feature Space Shen, Zhenyi Lu, Junru Gui, Lin Li, Jiazheng He, Yulan Yin, Di Sun, Xing Computation and Language Sparse attention reduces the quadratic complexity of full self-attention but faces two challenges: (1) an attention gap, where applying sparse attention to full-attention-trained models causes performance degradation due to train-inference distribution mismatch, and (2) a capability gap, where models trained purely with sparse attention lack complete gradient flow, preventing them from matching full-attention performance. We propose SSA (Sparse Sparse Attention), a training framework that integrates both sparse and full attention with bidirectional attention-output alignment. We prove that the approximation error scales linearly with the attention mass dropped under sparse attention, and show that SSA's alignment objective substantially reduces this quantity compared to baselines. Experiments demonstrate that SSA achieves state-of-the-art performance under both inference modes, adapts smoothly to varying sparsity budgets, and demonstrates superior long-context capabilities. The code is available at https://github.com/zhenyi4/ssa. |
| title | SSA: Sparse Sparse Attention by Aligning Full and Sparse Attention Outputs in Feature Space |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2511.20102 |