Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2026
|
| Sujets: | |
| Accès en ligne: | https://arxiv.org/abs/2603.15031 |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866912967904198656 |
|---|---|
| author | Kimi Team Chen, Guangyu Zhang, Yu Su, Jianlin Xu, Weixin Pan, Siyuan Wang, Yaoyu Wang, Yucheng Chen, Guanduo Yin, Bohong Chen, Yutian Yan, Junjie Wei, Ming Zhang, Y. Meng, Fanqing Hong, Chao Xie, Xiaotong Liu, Shaowei Lu, Enzhe Tai, Yunpeng Chen, Yanru Men, Xin Guo, Haiqing Charles, Y. Lu, Haoyu Sui, Lin Zhu, Jinguo Zhou, Zaida He, Weiran Huang, Weixiao Xu, Xinran Wang, Yuzhi Lai, Guokun Du, Yulun Wu, Yuxin Yang, Zhilin Zhou, Xinyu |
| author_facet | Kimi Team Chen, Guangyu Zhang, Yu Su, Jianlin Xu, Weixin Pan, Siyuan Wang, Yaoyu Wang, Yucheng Chen, Guanduo Yin, Bohong Chen, Yutian Yan, Junjie Wei, Ming Zhang, Y. Meng, Fanqing Hong, Chao Xie, Xiaotong Liu, Shaowei Lu, Enzhe Tai, Yunpeng Chen, Yanru Men, Xin Guo, Haiqing Charles, Y. Lu, Haoyu Sui, Lin Zhu, Jinguo Zhou, Zaida He, Weiran Huang, Weixiao Xu, Xinran Wang, Yuzhi Lai, Guokun Du, Yulun Wu, Yuxin Yang, Zhilin Zhou, Xinyu |
| contents | Residual connections with PreNorm are standard in modern LLMs, yet they accumulate all layer outputs with fixed unit weights. This uniform aggregation causes uncontrolled hidden-state growth with depth, progressively diluting each layer's contribution. We propose Attention Residuals (AttnRes), which replaces this fixed accumulation with softmax attention over preceding layer outputs, allowing each layer to selectively aggregate earlier representations with learned, input-dependent weights. To address the memory and communication overhead of attending over all preceding layer outputs for large-scale model training, we introduce Block AttnRes, which partitions layers into blocks and attends over block-level representations, reducing the memory footprint while preserving most of the gains of full AttnRes. Combined with cache-based pipeline communication and a two-phase computation strategy, Block AttnRes becomes a practical drop-in replacement for standard residual connections with minimal overhead.
Scaling law experiments confirm that the improvement is consistent across model sizes, and ablations validate the benefit of content-dependent depth-wise selection. We further integrate AttnRes into the Kimi Linear architecture (48B total / 3B activated parameters) and pre-train on 1.4T tokens, where AttnRes mitigates PreNorm dilution, yielding more uniform output magnitudes and gradient distribution across depth, and improves downstream performance across all evaluated tasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_15031 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Attention Residuals Kimi Team Chen, Guangyu Zhang, Yu Su, Jianlin Xu, Weixin Pan, Siyuan Wang, Yaoyu Wang, Yucheng Chen, Guanduo Yin, Bohong Chen, Yutian Yan, Junjie Wei, Ming Zhang, Y. Meng, Fanqing Hong, Chao Xie, Xiaotong Liu, Shaowei Lu, Enzhe Tai, Yunpeng Chen, Yanru Men, Xin Guo, Haiqing Charles, Y. Lu, Haoyu Sui, Lin Zhu, Jinguo Zhou, Zaida He, Weiran Huang, Weixiao Xu, Xinran Wang, Yuzhi Lai, Guokun Du, Yulun Wu, Yuxin Yang, Zhilin Zhou, Xinyu Computation and Language Residual connections with PreNorm are standard in modern LLMs, yet they accumulate all layer outputs with fixed unit weights. This uniform aggregation causes uncontrolled hidden-state growth with depth, progressively diluting each layer's contribution. We propose Attention Residuals (AttnRes), which replaces this fixed accumulation with softmax attention over preceding layer outputs, allowing each layer to selectively aggregate earlier representations with learned, input-dependent weights. To address the memory and communication overhead of attending over all preceding layer outputs for large-scale model training, we introduce Block AttnRes, which partitions layers into blocks and attends over block-level representations, reducing the memory footprint while preserving most of the gains of full AttnRes. Combined with cache-based pipeline communication and a two-phase computation strategy, Block AttnRes becomes a practical drop-in replacement for standard residual connections with minimal overhead. Scaling law experiments confirm that the improvement is consistent across model sizes, and ablations validate the benefit of content-dependent depth-wise selection. We further integrate AttnRes into the Kimi Linear architecture (48B total / 3B activated parameters) and pre-train on 1.4T tokens, where AttnRes mitigates PreNorm dilution, yielding more uniform output magnitudes and gradient distribution across depth, and improves downstream performance across all evaluated tasks. |
| title | Attention Residuals |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2603.15031 |