Sigma: Differential Rescaling of Query, Key and Value for Efficient Language Models
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915144280309760 |
|---|---|
| author | Lin, Zhenghao Tang, Zihao Liu, Xiao Gong, Yeyun Cheng, Yi Chen, Qi Li, Hang Xin, Ying Yang, Ziyue Yang, Kailai Yan, Yu Liang, Xiao Lu, Shuai Huang, Yiming Luo, Zheheng Qu, Lei Feng, Xuan Wang, Yaoxiang Xia, Yuqing Chen, Feiyang Jiang, Yuting Hu, Yasen Ni, Hao Li, Binyang Zhao, Guoshuai Chiang, Jui-Hao Guo, Zhongxin Lin, Chen Kuang, Kun Li, Wenjie Shen, Yelong Jiao, Jian Cheng, Peng Yang, Mao |
| author_facet | Lin, Zhenghao Tang, Zihao Liu, Xiao Gong, Yeyun Cheng, Yi Chen, Qi Li, Hang Xin, Ying Yang, Ziyue Yang, Kailai Yan, Yu Liang, Xiao Lu, Shuai Huang, Yiming Luo, Zheheng Qu, Lei Feng, Xuan Wang, Yaoxiang Xia, Yuqing Chen, Feiyang Jiang, Yuting Hu, Yasen Ni, Hao Li, Binyang Zhao, Guoshuai Chiang, Jui-Hao Guo, Zhongxin Lin, Chen Kuang, Kun Li, Wenjie Shen, Yelong Jiao, Jian Cheng, Peng Yang, Mao |
| contents | We introduce Sigma, an efficient large language model specialized for the system domain, empowered by a novel architecture including DiffQKV attention, and pre-trained on our meticulously collected system domain data. DiffQKV attention significantly enhances the inference efficiency of Sigma by optimizing the Query (Q), Key (K), and Value (V) components in the attention mechanism differentially, based on their varying impacts on the model performance and efficiency indicators. Specifically, we (1) conduct extensive experiments that demonstrate the model's varying sensitivity to the compression of K and V components, leading to the development of differentially compressed KV, and (2) propose augmented Q to expand the Q head dimension, which enhances the model's representation capacity with minimal impacts on the inference speed. Rigorous theoretical and empirical analyses reveal that DiffQKV attention significantly enhances efficiency, achieving up to a 33.36% improvement in inference speed over the conventional grouped-query attention (GQA) in long-context scenarios. We pre-train Sigma on 6T tokens from various sources, including 19.5B system domain data that we carefully collect and 1T tokens of synthesized and rewritten data. In general domains, Sigma achieves comparable performance to other state-of-arts models. In the system domain, we introduce the first comprehensive benchmark AIMicius, where Sigma demonstrates remarkable performance across all tasks, significantly outperforming GPT-4 with an absolute improvement up to 52.5%. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_13629 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Sigma: Differential Rescaling of Query, Key and Value for Efficient Language Models Lin, Zhenghao Tang, Zihao Liu, Xiao Gong, Yeyun Cheng, Yi Chen, Qi Li, Hang Xin, Ying Yang, Ziyue Yang, Kailai Yan, Yu Liang, Xiao Lu, Shuai Huang, Yiming Luo, Zheheng Qu, Lei Feng, Xuan Wang, Yaoxiang Xia, Yuqing Chen, Feiyang Jiang, Yuting Hu, Yasen Ni, Hao Li, Binyang Zhao, Guoshuai Chiang, Jui-Hao Guo, Zhongxin Lin, Chen Kuang, Kun Li, Wenjie Shen, Yelong Jiao, Jian Cheng, Peng Yang, Mao Computation and Language We introduce Sigma, an efficient large language model specialized for the system domain, empowered by a novel architecture including DiffQKV attention, and pre-trained on our meticulously collected system domain data. DiffQKV attention significantly enhances the inference efficiency of Sigma by optimizing the Query (Q), Key (K), and Value (V) components in the attention mechanism differentially, based on their varying impacts on the model performance and efficiency indicators. Specifically, we (1) conduct extensive experiments that demonstrate the model's varying sensitivity to the compression of K and V components, leading to the development of differentially compressed KV, and (2) propose augmented Q to expand the Q head dimension, which enhances the model's representation capacity with minimal impacts on the inference speed. Rigorous theoretical and empirical analyses reveal that DiffQKV attention significantly enhances efficiency, achieving up to a 33.36% improvement in inference speed over the conventional grouped-query attention (GQA) in long-context scenarios. We pre-train Sigma on 6T tokens from various sources, including 19.5B system domain data that we carefully collect and 1T tokens of synthesized and rewritten data. In general domains, Sigma achieves comparable performance to other state-of-arts models. In the system domain, we introduce the first comprehensive benchmark AIMicius, where Sigma demonstrates remarkable performance across all tasks, significantly outperforming GPT-4 with an absolute improvement up to 52.5%. |
| title | Sigma: Differential Rescaling of Query, Key and Value for Efficient Language Models |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2501.13629 |