Kimi Linear: An Expressive, Efficient Attention Architecture
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909881633603584 |
|---|---|
| author | Kimi Team Zhang, Yu Lin, Zongyu Yao, Xingcheng Hu, Jiaxi Meng, Fanqing Liu, Chengyin Men, Xin Yang, Songlin Li, Zhiyuan Li, Wentao Lu, Enzhe Liu, Weizhou Chen, Yanru Xu, Weixin Yu, Longhui Wang, Yejie Fan, Yu Zhong, Longguang Yuan, Enming Zhang, Dehao Zhang, Yizhi Liu, T. Y. Wang, Haiming Fang, Shengjun He, Weiran Liu, Shaowei Li, Yiwei Su, Jianlin Qiu, Jiezhong Pang, Bo Yan, Junjie Jiang, Zhejun Huang, Weixiao Yin, Bohong You, Jiacheng Wei, Chu Wang, Zhengtao Hong, Chao Chen, Yutian Chen, Guanduo Wang, Yucheng Zheng, Huabin Wang, Feng Liu, Yibo Dong, Mengnan Zhang, Zheng Pan, Siyuan Wu, Wenhao Wu, Yuhao Guan, Longyu Tao, Jiawen Fu, Guohong Xu, Xinran Wang, Yuzhi Lai, Guokun Wu, Yuxin Zhou, Xinyu Yang, Zhilin Du, Yulun |
| author_facet | Kimi Team Zhang, Yu Lin, Zongyu Yao, Xingcheng Hu, Jiaxi Meng, Fanqing Liu, Chengyin Men, Xin Yang, Songlin Li, Zhiyuan Li, Wentao Lu, Enzhe Liu, Weizhou Chen, Yanru Xu, Weixin Yu, Longhui Wang, Yejie Fan, Yu Zhong, Longguang Yuan, Enming Zhang, Dehao Zhang, Yizhi Liu, T. Y. Wang, Haiming Fang, Shengjun He, Weiran Liu, Shaowei Li, Yiwei Su, Jianlin Qiu, Jiezhong Pang, Bo Yan, Junjie Jiang, Zhejun Huang, Weixiao Yin, Bohong You, Jiacheng Wei, Chu Wang, Zhengtao Hong, Chao Chen, Yutian Chen, Guanduo Wang, Yucheng Zheng, Huabin Wang, Feng Liu, Yibo Dong, Mengnan Zhang, Zheng Pan, Siyuan Wu, Wenhao Wu, Yuhao Guan, Longyu Tao, Jiawen Fu, Guohong Xu, Xinran Wang, Yuzhi Lai, Guokun Wu, Yuxin Zhou, Xinyu Yang, Zhilin Du, Yulun |
| contents | We introduce Kimi Linear, a hybrid linear attention architecture that, for the first time, outperforms full attention under fair comparisons across various scenarios -- including short-context, long-context, and reinforcement learning (RL) scaling regimes. At its core lies Kimi Delta Attention (KDA), an expressive linear attention module that extends Gated DeltaNet with a finer-grained gating mechanism, enabling more effective use of limited finite-state RNN memory. Our bespoke chunkwise algorithm achieves high hardware efficiency through a specialized variant of the Diagonal-Plus-Low-Rank (DPLR) transition matrices, which substantially reduces computation compared to the general DPLR formulation while remaining more consistent with the classical delta rule.
We pretrain a Kimi Linear model with 3B activated parameters and 48B total parameters, based on a layerwise hybrid of KDA and Multi-Head Latent Attention (MLA). Our experiments show that with an identical training recipe, Kimi Linear outperforms full MLA with a sizeable margin across all evaluated tasks, while reducing KV cache usage by up to 75% and achieving up to 6 times decoding throughput for a 1M context. These results demonstrate that Kimi Linear can be a drop-in replacement for full attention architectures with superior performance and efficiency, including tasks with longer input and output lengths.
To support further research, we open-source the KDA kernel and vLLM implementations, and release the pre-trained and instruction-tuned model checkpoints. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_26692 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Kimi Linear: An Expressive, Efficient Attention Architecture Kimi Team Zhang, Yu Lin, Zongyu Yao, Xingcheng Hu, Jiaxi Meng, Fanqing Liu, Chengyin Men, Xin Yang, Songlin Li, Zhiyuan Li, Wentao Lu, Enzhe Liu, Weizhou Chen, Yanru Xu, Weixin Yu, Longhui Wang, Yejie Fan, Yu Zhong, Longguang Yuan, Enming Zhang, Dehao Zhang, Yizhi Liu, T. Y. Wang, Haiming Fang, Shengjun He, Weiran Liu, Shaowei Li, Yiwei Su, Jianlin Qiu, Jiezhong Pang, Bo Yan, Junjie Jiang, Zhejun Huang, Weixiao Yin, Bohong You, Jiacheng Wei, Chu Wang, Zhengtao Hong, Chao Chen, Yutian Chen, Guanduo Wang, Yucheng Zheng, Huabin Wang, Feng Liu, Yibo Dong, Mengnan Zhang, Zheng Pan, Siyuan Wu, Wenhao Wu, Yuhao Guan, Longyu Tao, Jiawen Fu, Guohong Xu, Xinran Wang, Yuzhi Lai, Guokun Wu, Yuxin Zhou, Xinyu Yang, Zhilin Du, Yulun Computation and Language Machine Learning We introduce Kimi Linear, a hybrid linear attention architecture that, for the first time, outperforms full attention under fair comparisons across various scenarios -- including short-context, long-context, and reinforcement learning (RL) scaling regimes. At its core lies Kimi Delta Attention (KDA), an expressive linear attention module that extends Gated DeltaNet with a finer-grained gating mechanism, enabling more effective use of limited finite-state RNN memory. Our bespoke chunkwise algorithm achieves high hardware efficiency through a specialized variant of the Diagonal-Plus-Low-Rank (DPLR) transition matrices, which substantially reduces computation compared to the general DPLR formulation while remaining more consistent with the classical delta rule. We pretrain a Kimi Linear model with 3B activated parameters and 48B total parameters, based on a layerwise hybrid of KDA and Multi-Head Latent Attention (MLA). Our experiments show that with an identical training recipe, Kimi Linear outperforms full MLA with a sizeable margin across all evaluated tasks, while reducing KV cache usage by up to 75% and achieving up to 6 times decoding throughput for a 1M context. These results demonstrate that Kimi Linear can be a drop-in replacement for full attention architectures with superior performance and efficiency, including tasks with longer input and output lengths. To support further research, we open-source the KDA kernel and vLLM implementations, and release the pre-trained and instruction-tuned model checkpoints. |
| title | Kimi Linear: An Expressive, Efficient Attention Architecture |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2510.26692 |