DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866911925734998016 |
|---|---|
| author | DeepSeek-AI Liu, Aixin Feng, Bei Wang, Bin Wang, Bingxuan Liu, Bo Zhao, Chenggang Dengr, Chengqi Ruan, Chong Dai, Damai Guo, Daya Yang, Dejian Chen, Deli Ji, Dongjie Li, Erhang Lin, Fangyun Luo, Fuli Hao, Guangbo Chen, Guanting Li, Guowei Zhang, H. Xu, Hanwei Yang, Hao Zhang, Haowei Ding, Honghui Xin, Huajian Gao, Huazuo Li, Hui Qu, Hui Cai, J. L. Liang, Jian Guo, Jianzhong Ni, Jiaqi Li, Jiashi Chen, Jin Yuan, Jingyang Qiu, Junjie Song, Junxiao Dong, Kai Gao, Kaige Guan, Kang Wang, Lean Zhang, Lecong Xu, Lei Xia, Leyi Zhao, Liang Zhang, Liyue Li, Meng Wang, Miaojun Zhang, Mingchuan Zhang, Minghua Tang, Minghui Li, Mingming Tian, Ning Huang, Panpan Wang, Peiyi Zhang, Peng Zhu, Qihao Chen, Qinyu Du, Qiushi Chen, R. J. Jin, R. L. Ge, Ruiqi Pan, Ruizhe Xu, Runxin Chen, Ruyi Li, S. S. Lu, Shanghao Zhou, Shangyan Chen, Shanhuang Wu, Shaoqing Ye, Shengfeng Ma, Shirong Wang, Shiyu Zhou, Shuang Yu, Shuiping Zhou, Shunfeng Zheng, Size Wang, T. Pei, Tian Yuan, Tian Sun, Tianyu Xiao, W. L. Zeng, Wangding An, Wei Liu, Wen Liang, Wenfeng Gao, Wenjun Zhang, Wentao Li, X. Q. Jin, Xiangyue Wang, Xianzu Bi, Xiao Liu, Xiaodong Wang, Xiaohan Shen, Xiaojin Chen, Xiaokang Chen, Xiaosha Nie, Xiaotao Sun, Xiaowen Wang, Xiaoxiang Liu, Xin Xie, Xin Yu, Xingkai Song, Xinnan Zhou, Xinyi Yang, Xinyu Lu, Xuan Su, Xuecheng Wu, Y. Li, Y. K. Wei, Y. X. Zhu, Y. X. Xu, Yanhong Huang, Yanping Li, Yao Zhao, Yao Sun, Yaofeng Li, Yaohui Wang, Yaohui Zheng, Yi Zhang, Yichao Xiong, Yiliang Zhao, Yilong He, Ying Tang, Ying Piao, Yishi Dong, Yixin Tan, Yixuan Liu, Yiyuan Wang, Yongji Guo, Yongqiang Zhu, Yuchen Wang, Yuduan Zou, Yuheng Zha, Yukun Ma, Yunxian Yan, Yuting You, Yuxiang Liu, Yuxuan Ren, Z. Z. Ren, Zehui Sha, Zhangli Fu, Zhe Huang, Zhen Zhang, Zhen Xie, Zhenda Hao, Zhewen Shao, Zhihong Wen, Zhiniu Xu, Zhipeng Zhang, Zhongyu Li, Zhuoshu Wang, Zihan Gu, Zihui Li, Zilin Xie, Ziwei |
| author_facet | DeepSeek-AI Liu, Aixin Feng, Bei Wang, Bin Wang, Bingxuan Liu, Bo Zhao, Chenggang Dengr, Chengqi Ruan, Chong Dai, Damai Guo, Daya Yang, Dejian Chen, Deli Ji, Dongjie Li, Erhang Lin, Fangyun Luo, Fuli Hao, Guangbo Chen, Guanting Li, Guowei Zhang, H. Xu, Hanwei Yang, Hao Zhang, Haowei Ding, Honghui Xin, Huajian Gao, Huazuo Li, Hui Qu, Hui Cai, J. L. Liang, Jian Guo, Jianzhong Ni, Jiaqi Li, Jiashi Chen, Jin Yuan, Jingyang Qiu, Junjie Song, Junxiao Dong, Kai Gao, Kaige Guan, Kang Wang, Lean Zhang, Lecong Xu, Lei Xia, Leyi Zhao, Liang Zhang, Liyue Li, Meng Wang, Miaojun Zhang, Mingchuan Zhang, Minghua Tang, Minghui Li, Mingming Tian, Ning Huang, Panpan Wang, Peiyi Zhang, Peng Zhu, Qihao Chen, Qinyu Du, Qiushi Chen, R. J. Jin, R. L. Ge, Ruiqi Pan, Ruizhe Xu, Runxin Chen, Ruyi Li, S. S. Lu, Shanghao Zhou, Shangyan Chen, Shanhuang Wu, Shaoqing Ye, Shengfeng Ma, Shirong Wang, Shiyu Zhou, Shuang Yu, Shuiping Zhou, Shunfeng Zheng, Size Wang, T. Pei, Tian Yuan, Tian Sun, Tianyu Xiao, W. L. Zeng, Wangding An, Wei Liu, Wen Liang, Wenfeng Gao, Wenjun Zhang, Wentao Li, X. Q. Jin, Xiangyue Wang, Xianzu Bi, Xiao Liu, Xiaodong Wang, Xiaohan Shen, Xiaojin Chen, Xiaokang Chen, Xiaosha Nie, Xiaotao Sun, Xiaowen Wang, Xiaoxiang Liu, Xin Xie, Xin Yu, Xingkai Song, Xinnan Zhou, Xinyi Yang, Xinyu Lu, Xuan Su, Xuecheng Wu, Y. Li, Y. K. Wei, Y. X. Zhu, Y. X. Xu, Yanhong Huang, Yanping Li, Yao Zhao, Yao Sun, Yaofeng Li, Yaohui Wang, Yaohui Zheng, Yi Zhang, Yichao Xiong, Yiliang Zhao, Yilong He, Ying Tang, Ying Piao, Yishi Dong, Yixin Tan, Yixuan Liu, Yiyuan Wang, Yongji Guo, Yongqiang Zhu, Yuchen Wang, Yuduan Zou, Yuheng Zha, Yukun Ma, Yunxian Yan, Yuting You, Yuxiang Liu, Yuxuan Ren, Z. Z. Ren, Zehui Sha, Zhangli Fu, Zhe Huang, Zhen Zhang, Zhen Xie, Zhenda Hao, Zhewen Shao, Zhihong Wen, Zhiniu Xu, Zhipeng Zhang, Zhongyu Li, Zhuoshu Wang, Zihan Gu, Zihui Li, Zilin Xie, Ziwei |
| contents | We present DeepSeek-V2, a strong Mixture-of-Experts (MoE) language model characterized by economical training and efficient inference. It comprises 236B total parameters, of which 21B are activated for each token, and supports a context length of 128K tokens. DeepSeek-V2 adopts innovative architectures including Multi-head Latent Attention (MLA) and DeepSeekMoE. MLA guarantees efficient inference through significantly compressing the Key-Value (KV) cache into a latent vector, while DeepSeekMoE enables training strong models at an economical cost through sparse computation. Compared with DeepSeek 67B, DeepSeek-V2 achieves significantly stronger performance, and meanwhile saves 42.5% of training costs, reduces the KV cache by 93.3%, and boosts the maximum generation throughput to 5.76 times. We pretrain DeepSeek-V2 on a high-quality and multi-source corpus consisting of 8.1T tokens, and further perform Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) to fully unlock its potential. Evaluation results show that, even with only 21B activated parameters, DeepSeek-V2 and its chat versions still achieve top-tier performance among open-source models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_04434 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model DeepSeek-AI Liu, Aixin Feng, Bei Wang, Bin Wang, Bingxuan Liu, Bo Zhao, Chenggang Dengr, Chengqi Ruan, Chong Dai, Damai Guo, Daya Yang, Dejian Chen, Deli Ji, Dongjie Li, Erhang Lin, Fangyun Luo, Fuli Hao, Guangbo Chen, Guanting Li, Guowei Zhang, H. Xu, Hanwei Yang, Hao Zhang, Haowei Ding, Honghui Xin, Huajian Gao, Huazuo Li, Hui Qu, Hui Cai, J. L. Liang, Jian Guo, Jianzhong Ni, Jiaqi Li, Jiashi Chen, Jin Yuan, Jingyang Qiu, Junjie Song, Junxiao Dong, Kai Gao, Kaige Guan, Kang Wang, Lean Zhang, Lecong Xu, Lei Xia, Leyi Zhao, Liang Zhang, Liyue Li, Meng Wang, Miaojun Zhang, Mingchuan Zhang, Minghua Tang, Minghui Li, Mingming Tian, Ning Huang, Panpan Wang, Peiyi Zhang, Peng Zhu, Qihao Chen, Qinyu Du, Qiushi Chen, R. J. Jin, R. L. Ge, Ruiqi Pan, Ruizhe Xu, Runxin Chen, Ruyi Li, S. S. Lu, Shanghao Zhou, Shangyan Chen, Shanhuang Wu, Shaoqing Ye, Shengfeng Ma, Shirong Wang, Shiyu Zhou, Shuang Yu, Shuiping Zhou, Shunfeng Zheng, Size Wang, T. Pei, Tian Yuan, Tian Sun, Tianyu Xiao, W. L. Zeng, Wangding An, Wei Liu, Wen Liang, Wenfeng Gao, Wenjun Zhang, Wentao Li, X. Q. Jin, Xiangyue Wang, Xianzu Bi, Xiao Liu, Xiaodong Wang, Xiaohan Shen, Xiaojin Chen, Xiaokang Chen, Xiaosha Nie, Xiaotao Sun, Xiaowen Wang, Xiaoxiang Liu, Xin Xie, Xin Yu, Xingkai Song, Xinnan Zhou, Xinyi Yang, Xinyu Lu, Xuan Su, Xuecheng Wu, Y. Li, Y. K. Wei, Y. X. Zhu, Y. X. Xu, Yanhong Huang, Yanping Li, Yao Zhao, Yao Sun, Yaofeng Li, Yaohui Wang, Yaohui Zheng, Yi Zhang, Yichao Xiong, Yiliang Zhao, Yilong He, Ying Tang, Ying Piao, Yishi Dong, Yixin Tan, Yixuan Liu, Yiyuan Wang, Yongji Guo, Yongqiang Zhu, Yuchen Wang, Yuduan Zou, Yuheng Zha, Yukun Ma, Yunxian Yan, Yuting You, Yuxiang Liu, Yuxuan Ren, Z. Z. Ren, Zehui Sha, Zhangli Fu, Zhe Huang, Zhen Zhang, Zhen Xie, Zhenda Hao, Zhewen Shao, Zhihong Wen, Zhiniu Xu, Zhipeng Zhang, Zhongyu Li, Zhuoshu Wang, Zihan Gu, Zihui Li, Zilin Xie, Ziwei Computation and Language Artificial Intelligence We present DeepSeek-V2, a strong Mixture-of-Experts (MoE) language model characterized by economical training and efficient inference. It comprises 236B total parameters, of which 21B are activated for each token, and supports a context length of 128K tokens. DeepSeek-V2 adopts innovative architectures including Multi-head Latent Attention (MLA) and DeepSeekMoE. MLA guarantees efficient inference through significantly compressing the Key-Value (KV) cache into a latent vector, while DeepSeekMoE enables training strong models at an economical cost through sparse computation. Compared with DeepSeek 67B, DeepSeek-V2 achieves significantly stronger performance, and meanwhile saves 42.5% of training costs, reduces the KV cache by 93.3%, and boosts the maximum generation throughput to 5.76 times. We pretrain DeepSeek-V2 on a high-quality and multi-source corpus consisting of 8.1T tokens, and further perform Supervised Fine-Tuning (SFT) and Reinforcement Learning (RL) to fully unlock its potential. Evaluation results show that, even with only 21B activated parameters, DeepSeek-V2 and its chat versions still achieve top-tier performance among open-source models. |
| title | DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Model |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2405.04434 |