SageAttention2++: A More Efficient Implementation of SageAttention2
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866910991316418560 |
|---|---|
| author | Zhang, Jintao Xu, Xiaoming Wei, Jia Huang, Haofeng Zhang, Pengle Xiang, Chendong Zhu, Jun Chen, Jianfei |
| author_facet | Zhang, Jintao Xu, Xiaoming Wei, Jia Huang, Haofeng Zhang, Pengle Xiang, Chendong Zhu, Jun Chen, Jianfei |
| contents | The efficiency of attention is critical because its time complexity grows quadratically with sequence length. SageAttention2 addresses this by utilizing quantization to accelerate matrix multiplications (Matmul) in attention. To further accelerate SageAttention2, we propose to utilize the faster instruction of FP8 Matmul accumulated in FP16. The instruction is 2x faster than the FP8 Matmul used in SageAttention2. Our experiments show that SageAttention2++ achieves a 3.9x speedup over FlashAttention while maintaining the same attention accuracy as SageAttention2. This means SageAttention2++ effectively accelerates various models, including those for language, image, and video generation, with negligible end-to-end metrics loss. The code will be available at https://github.com/thu-ml/SageAttention. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_21136 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | SageAttention2++: A More Efficient Implementation of SageAttention2 Zhang, Jintao Xu, Xiaoming Wei, Jia Huang, Haofeng Zhang, Pengle Xiang, Chendong Zhu, Jun Chen, Jianfei Machine Learning Artificial Intelligence Hardware Architecture Computer Vision and Pattern Recognition The efficiency of attention is critical because its time complexity grows quadratically with sequence length. SageAttention2 addresses this by utilizing quantization to accelerate matrix multiplications (Matmul) in attention. To further accelerate SageAttention2, we propose to utilize the faster instruction of FP8 Matmul accumulated in FP16. The instruction is 2x faster than the FP8 Matmul used in SageAttention2. Our experiments show that SageAttention2++ achieves a 3.9x speedup over FlashAttention while maintaining the same attention accuracy as SageAttention2. This means SageAttention2++ effectively accelerates various models, including those for language, image, and video generation, with negligible end-to-end metrics loss. The code will be available at https://github.com/thu-ml/SageAttention. |
| title | SageAttention2++: A More Efficient Implementation of SageAttention2 |
| topic | Machine Learning Artificial Intelligence Hardware Architecture Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2505.21136 |