MiniCPM-SALA: Hybridizing Sparse and Linear Attention for Efficient Long-Context Modeling
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2026
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866915824137142272 |
|---|---|
| author | MiniCPM Team An, Wenhao Chen, Yingfa Fang, Yewei Li, Jiayi Li, Xin Li, Yaohui Li, Yishan Li, Yuxuan Lin, Biyuan Liu, Chuan Liu, Hezi Liu, Siyuan Lyu, Hongya Pan, Yinxu Ren, Shixin Shen, Xingyu Su, Zhou Sun, Haojun Sun, Yangang Thai, Zhen Leng Tian, Xin Wang, Rui Wang, Xiaorong Wang, Yudong Wu, Bo Xu, Xiaoyue Xu, Dong Xue, Shuaikang Yang, Jiawei Zhang, Bowen Zhang, Jinqian Zhang, Letian Zhang, Shengnan Zhang, Xinyu Zhang, Xinyuan Zhang, Zhu Zhao, Hengyu Zhao, Jiacheng Zheng, Zhi Zhou, Jie Zhou, Zihan Wang, Shuo Xiao, Chaojun Han, Xu Liu, Zhiyuan Sun, Maosong |
| author_facet | MiniCPM Team An, Wenhao Chen, Yingfa Fang, Yewei Li, Jiayi Li, Xin Li, Yaohui Li, Yishan Li, Yuxuan Lin, Biyuan Liu, Chuan Liu, Hezi Liu, Siyuan Lyu, Hongya Pan, Yinxu Ren, Shixin Shen, Xingyu Su, Zhou Sun, Haojun Sun, Yangang Thai, Zhen Leng Tian, Xin Wang, Rui Wang, Xiaorong Wang, Yudong Wu, Bo Xu, Xiaoyue Xu, Dong Xue, Shuaikang Yang, Jiawei Zhang, Bowen Zhang, Jinqian Zhang, Letian Zhang, Shengnan Zhang, Xinyu Zhang, Xinyuan Zhang, Zhu Zhao, Hengyu Zhao, Jiacheng Zheng, Zhi Zhou, Jie Zhou, Zihan Wang, Shuo Xiao, Chaojun Han, Xu Liu, Zhiyuan Sun, Maosong |
| contents | The evolution of large language models (LLMs) towards applications with ultra-long contexts faces challenges posed by the high computational and memory costs of the Transformer architecture. While existing sparse and linear attention mechanisms attempt to mitigate these issues, they typically involve a trade-off between memory efficiency and model performance. This paper introduces MiniCPM-SALA, a 9B-parameter hybrid architecture that integrates the high-fidelity long-context modeling of sparse attention (InfLLM-V2) with the global efficiency of linear attention (Lightning Attention). By employing a layer selection algorithm to integrate these mechanisms in a 1:3 ratio and utilizing a hybrid positional encoding (HyPE), the model maintains efficiency and performance for long-context tasks. Furthermore, we introduce a cost-effective continual training framework that transforms pre-trained Transformer-based models into hybrid models, which reduces training costs by approximately 75% compared to training from scratch. Extensive experiments show that MiniCPM-SALA maintains general capabilities comparable to full-attention models while offering improved efficiency. On a single NVIDIA A6000D GPU, the model achieves up to 3.5x the inference speed of the full-attention model at the sequence length of 256K tokens and supports context lengths of up to 1M tokens, a scale where traditional full-attention 8B models fail because of memory constraints. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_11761 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | MiniCPM-SALA: Hybridizing Sparse and Linear Attention for Efficient Long-Context Modeling MiniCPM Team An, Wenhao Chen, Yingfa Fang, Yewei Li, Jiayi Li, Xin Li, Yaohui Li, Yishan Li, Yuxuan Lin, Biyuan Liu, Chuan Liu, Hezi Liu, Siyuan Lyu, Hongya Pan, Yinxu Ren, Shixin Shen, Xingyu Su, Zhou Sun, Haojun Sun, Yangang Thai, Zhen Leng Tian, Xin Wang, Rui Wang, Xiaorong Wang, Yudong Wu, Bo Xu, Xiaoyue Xu, Dong Xue, Shuaikang Yang, Jiawei Zhang, Bowen Zhang, Jinqian Zhang, Letian Zhang, Shengnan Zhang, Xinyu Zhang, Xinyuan Zhang, Zhu Zhao, Hengyu Zhao, Jiacheng Zheng, Zhi Zhou, Jie Zhou, Zihan Wang, Shuo Xiao, Chaojun Han, Xu Liu, Zhiyuan Sun, Maosong Computation and Language Artificial Intelligence Machine Learning The evolution of large language models (LLMs) towards applications with ultra-long contexts faces challenges posed by the high computational and memory costs of the Transformer architecture. While existing sparse and linear attention mechanisms attempt to mitigate these issues, they typically involve a trade-off between memory efficiency and model performance. This paper introduces MiniCPM-SALA, a 9B-parameter hybrid architecture that integrates the high-fidelity long-context modeling of sparse attention (InfLLM-V2) with the global efficiency of linear attention (Lightning Attention). By employing a layer selection algorithm to integrate these mechanisms in a 1:3 ratio and utilizing a hybrid positional encoding (HyPE), the model maintains efficiency and performance for long-context tasks. Furthermore, we introduce a cost-effective continual training framework that transforms pre-trained Transformer-based models into hybrid models, which reduces training costs by approximately 75% compared to training from scratch. Extensive experiments show that MiniCPM-SALA maintains general capabilities comparable to full-attention models while offering improved efficiency. On a single NVIDIA A6000D GPU, the model achieves up to 3.5x the inference speed of the full-attention model at the sequence length of 256K tokens and supports context lengths of up to 1M tokens, a scale where traditional full-attention 8B models fail because of memory constraints. |
| title | MiniCPM-SALA: Hybridizing Sparse and Linear Attention for Efficient Long-Context Modeling |
| topic | Computation and Language Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2602.11761 |