SLA2: Sparse-Linear Attention with Learnable Routing and QAT
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2026
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866910021131960320 |
|---|---|
| author | Zhang, Jintao Wang, Haoxu Jiang, Kai Zheng, Kaiwen Jiang, Youhe Stoica, Ion Chen, Jianfei Zhu, Jun Gonzalez, Joseph E. |
| author_facet | Zhang, Jintao Wang, Haoxu Jiang, Kai Zheng, Kaiwen Jiang, Youhe Stoica, Ion Chen, Jianfei Zhu, Jun Gonzalez, Joseph E. |
| contents | Sparse-Linear Attention (SLA) combines sparse and linear attention to accelerate diffusion models and has shown strong performance in video generation. However, (i) SLA relies on a heuristic split that assigns computations to the sparse or linear branch based on attention-weight magnitude, which can be suboptimal. Additionally, (ii) after formally analyzing the attention error in SLA, we identify a mismatch between SLA and a direct decomposition into sparse and linear attention. We propose SLA2, which introduces (I) a learnable router that dynamically selects whether each attention computation should use sparse or linear attention, (II) a more faithful and direct sparse-linear attention formulation that uses a learnable ratio to combine the sparse and linear attention branches, and (III) a sparse + low-bit attention design, where low-bit attention is introduced via quantization-aware fine-tuning to reduce quantization error. Experiments show that on video diffusion models, SLA2 can achieve 97% attention sparsity and deliver an 18.6x attention speedup while preserving generation quality. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_12675 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | SLA2: Sparse-Linear Attention with Learnable Routing and QAT Zhang, Jintao Wang, Haoxu Jiang, Kai Zheng, Kaiwen Jiang, Youhe Stoica, Ion Chen, Jianfei Zhu, Jun Gonzalez, Joseph E. Machine Learning Artificial Intelligence Computer Vision and Pattern Recognition Sparse-Linear Attention (SLA) combines sparse and linear attention to accelerate diffusion models and has shown strong performance in video generation. However, (i) SLA relies on a heuristic split that assigns computations to the sparse or linear branch based on attention-weight magnitude, which can be suboptimal. Additionally, (ii) after formally analyzing the attention error in SLA, we identify a mismatch between SLA and a direct decomposition into sparse and linear attention. We propose SLA2, which introduces (I) a learnable router that dynamically selects whether each attention computation should use sparse or linear attention, (II) a more faithful and direct sparse-linear attention formulation that uses a learnable ratio to combine the sparse and linear attention branches, and (III) a sparse + low-bit attention design, where low-bit attention is introduced via quantization-aware fine-tuning to reduce quantization error. Experiments show that on video diffusion models, SLA2 can achieve 97% attention sparsity and deliver an 18.6x attention speedup while preserving generation quality. |
| title | SLA2: Sparse-Linear Attention with Learnable Routing and QAT |
| topic | Machine Learning Artificial Intelligence Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2602.12675 |