Kwai Summary Attention Technical Report
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2026
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866914510614298624 |
|---|---|
| author | Chu, Chenglong Zhou, Guorui Zhang, Guowang Li, Han Peng, Hao Cheng, Hongtao Liang, Jian Cao, Jiangxia Gai, Kun Zhou, Lingzhi Ren, Lu Zhang, Qi Tang, Ruiming Wang, Ruitao Luo, Xinchen Su, Yi Liang, Zhiyuan Wang, Ziqi Ding, Boyang Song, Chengru Zang, Dunju Wang, Hui Ou, Jiao Deng, Jiaxin Shi, Jijun Zhang, Jinghao Chen, Junmin Ren, Lejian Lv, Minxuan Wang, Qianqian Hu, Qigen Wang, Shiyao Mao, Siyang Wang, Tao Wang, Xingmei Ling, Zhixin Li, Ziming Zhang, Zixing |
| author_facet | Chu, Chenglong Zhou, Guorui Zhang, Guowang Li, Han Peng, Hao Cheng, Hongtao Liang, Jian Cao, Jiangxia Gai, Kun Zhou, Lingzhi Ren, Lu Zhang, Qi Tang, Ruiming Wang, Ruitao Luo, Xinchen Su, Yi Liang, Zhiyuan Wang, Ziqi Ding, Boyang Song, Chengru Zang, Dunju Wang, Hui Ou, Jiao Deng, Jiaxin Shi, Jijun Zhang, Jinghao Chen, Junmin Ren, Lejian Lv, Minxuan Wang, Qianqian Hu, Qigen Wang, Shiyao Mao, Siyang Wang, Tao Wang, Xingmei Ling, Zhixin Li, Ziming Zhang, Zixing |
| contents | Long-context ability, has become one of the most important iteration direction of next-generation Large Language Models, particularly in semantic understanding/reasoning, code agentic intelligence and recommendation system. However, the standard softmax attention exhibits quadratic time complexity with respect to sequence length. As the sequence length increases, this incurs substantial overhead in long-context settings, leading the training and inference costs of extremely long sequences deteriorate rapidly. Existing solutions mitigate this issue through two technique routings: i) Reducing the KV cache per layer, such as from the head-level compression GQA, and the embedding dimension-level compression MLA, but the KV cache remains linearly dependent on the sequence length at a 1:1 ratio. ii) Interleaving with KV Cache friendly architecture, such as local attention SWA, linear kernel GDN, but often involve trade-offs among KV Cache and long-context modeling effectiveness. Besides the two technique routings, we argue that there exists an intermediate path not well explored: {Maintaining a linear relationship between the KV cache and sequence length, but performing semantic-level compression through a specific ratio $k$}. This $O(n/k)$ path does not pursue a ``minimum KV cache'', but rather trades acceptable memory costs for complete, referential, and interpretable retention of long distant dependency. Motivated by this, we propose Kwai Summary Attention (KSA), a novel attention mechanism that reduces sequence modeling cost by compressing historical contexts into learnable summary tokens. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_24432 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Kwai Summary Attention Technical Report Chu, Chenglong Zhou, Guorui Zhang, Guowang Li, Han Peng, Hao Cheng, Hongtao Liang, Jian Cao, Jiangxia Gai, Kun Zhou, Lingzhi Ren, Lu Zhang, Qi Tang, Ruiming Wang, Ruitao Luo, Xinchen Su, Yi Liang, Zhiyuan Wang, Ziqi Ding, Boyang Song, Chengru Zang, Dunju Wang, Hui Ou, Jiao Deng, Jiaxin Shi, Jijun Zhang, Jinghao Chen, Junmin Ren, Lejian Lv, Minxuan Wang, Qianqian Hu, Qigen Wang, Shiyao Mao, Siyang Wang, Tao Wang, Xingmei Ling, Zhixin Li, Ziming Zhang, Zixing Computation and Language Artificial Intelligence Information Retrieval Machine Learning Long-context ability, has become one of the most important iteration direction of next-generation Large Language Models, particularly in semantic understanding/reasoning, code agentic intelligence and recommendation system. However, the standard softmax attention exhibits quadratic time complexity with respect to sequence length. As the sequence length increases, this incurs substantial overhead in long-context settings, leading the training and inference costs of extremely long sequences deteriorate rapidly. Existing solutions mitigate this issue through two technique routings: i) Reducing the KV cache per layer, such as from the head-level compression GQA, and the embedding dimension-level compression MLA, but the KV cache remains linearly dependent on the sequence length at a 1:1 ratio. ii) Interleaving with KV Cache friendly architecture, such as local attention SWA, linear kernel GDN, but often involve trade-offs among KV Cache and long-context modeling effectiveness. Besides the two technique routings, we argue that there exists an intermediate path not well explored: {Maintaining a linear relationship between the KV cache and sequence length, but performing semantic-level compression through a specific ratio $k$}. This $O(n/k)$ path does not pursue a ``minimum KV cache'', but rather trades acceptable memory costs for complete, referential, and interpretable retention of long distant dependency. Motivated by this, we propose Kwai Summary Attention (KSA), a novel attention mechanism that reduces sequence modeling cost by compressing historical contexts into learnable summary tokens. |
| title | Kwai Summary Attention Technical Report |
| topic | Computation and Language Artificial Intelligence Information Retrieval Machine Learning |
| url | https://arxiv.org/abs/2604.24432 |