Pangu Ultra: Pushing the Limits of Dense Large Language Models on Ascend NPUs
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866908313096028160 |
|---|---|
| author | Yin, Yichun Huang, Wenyong Song, Kaikai Tang, Yehui Wu, Xueyu Guo, Wei Guo, Peng Wang, Yaoyuan Meng, Xiaojun Wang, Yasheng Li, Dong Chen, Can Tu, Dandan Li, Yin Yu, Fisher Tang, Ruiming Wang, Yunhe Wang, Baojun Wang, Bin Wang, Bo Liu, Boxiao Zhang, Changzheng Tang, Duyu Mi, Fei Jin, Hui Wei, Jiansheng Qin, Jiarui Li, Jinpeng Zhao, Jun Deng, Liqun Li, Lin Xu, Minghui Zhang, Naifu Zheng, Nianzu Li, Qiang Ruan, Rongju Cheng, Shengjun Guo, Tianyu He, Wei Li, Wei Liu, Weiwen Liu, Wulong Dai, Xinyi Dong, Yonghan Pan, Yu Li, Yue Wang, Yufei Li, Yujun Ni, Yunsheng Liu, Zhe Zhang, Zhenhe Liu, Zhicheng |
| author_facet | Yin, Yichun Huang, Wenyong Song, Kaikai Tang, Yehui Wu, Xueyu Guo, Wei Guo, Peng Wang, Yaoyuan Meng, Xiaojun Wang, Yasheng Li, Dong Chen, Can Tu, Dandan Li, Yin Yu, Fisher Tang, Ruiming Wang, Yunhe Wang, Baojun Wang, Bin Wang, Bo Liu, Boxiao Zhang, Changzheng Tang, Duyu Mi, Fei Jin, Hui Wei, Jiansheng Qin, Jiarui Li, Jinpeng Zhao, Jun Deng, Liqun Li, Lin Xu, Minghui Zhang, Naifu Zheng, Nianzu Li, Qiang Ruan, Rongju Cheng, Shengjun Guo, Tianyu He, Wei Li, Wei Liu, Weiwen Liu, Wulong Dai, Xinyi Dong, Yonghan Pan, Yu Li, Yue Wang, Yufei Li, Yujun Ni, Yunsheng Liu, Zhe Zhang, Zhenhe Liu, Zhicheng |
| contents | We present Pangu Ultra, a Large Language Model (LLM) with 135 billion parameters and dense Transformer modules trained on Ascend Neural Processing Units (NPUs). Although the field of LLM has been witnessing unprecedented advances in pushing the scale and capability of LLM in recent years, training such a large-scale model still involves significant optimization and system challenges. To stabilize the training process, we propose depth-scaled sandwich normalization, which effectively eliminates loss spikes during the training process of deep models. We pre-train our model on 13.2 trillion diverse and high-quality tokens and further enhance its reasoning capabilities during post-training. To perform such large-scale training efficiently, we utilize 8,192 Ascend NPUs with a series of system optimizations. Evaluations on multiple diverse benchmarks indicate that Pangu Ultra significantly advances the state-of-the-art capabilities of dense LLMs such as Llama 405B and Mistral Large 2, and even achieves competitive results with DeepSeek-R1, whose sparse model structure contains much more parameters. Our exploration demonstrates that Ascend NPUs are capable of efficiently and effectively training dense models with more than 100 billion parameters. Our model and system will be available for our commercial customers. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_07866 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Pangu Ultra: Pushing the Limits of Dense Large Language Models on Ascend NPUs Yin, Yichun Huang, Wenyong Song, Kaikai Tang, Yehui Wu, Xueyu Guo, Wei Guo, Peng Wang, Yaoyuan Meng, Xiaojun Wang, Yasheng Li, Dong Chen, Can Tu, Dandan Li, Yin Yu, Fisher Tang, Ruiming Wang, Yunhe Wang, Baojun Wang, Bin Wang, Bo Liu, Boxiao Zhang, Changzheng Tang, Duyu Mi, Fei Jin, Hui Wei, Jiansheng Qin, Jiarui Li, Jinpeng Zhao, Jun Deng, Liqun Li, Lin Xu, Minghui Zhang, Naifu Zheng, Nianzu Li, Qiang Ruan, Rongju Cheng, Shengjun Guo, Tianyu He, Wei Li, Wei Liu, Weiwen Liu, Wulong Dai, Xinyi Dong, Yonghan Pan, Yu Li, Yue Wang, Yufei Li, Yujun Ni, Yunsheng Liu, Zhe Zhang, Zhenhe Liu, Zhicheng Computation and Language Artificial Intelligence We present Pangu Ultra, a Large Language Model (LLM) with 135 billion parameters and dense Transformer modules trained on Ascend Neural Processing Units (NPUs). Although the field of LLM has been witnessing unprecedented advances in pushing the scale and capability of LLM in recent years, training such a large-scale model still involves significant optimization and system challenges. To stabilize the training process, we propose depth-scaled sandwich normalization, which effectively eliminates loss spikes during the training process of deep models. We pre-train our model on 13.2 trillion diverse and high-quality tokens and further enhance its reasoning capabilities during post-training. To perform such large-scale training efficiently, we utilize 8,192 Ascend NPUs with a series of system optimizations. Evaluations on multiple diverse benchmarks indicate that Pangu Ultra significantly advances the state-of-the-art capabilities of dense LLMs such as Llama 405B and Mistral Large 2, and even achieves competitive results with DeepSeek-R1, whose sparse model structure contains much more parameters. Our exploration demonstrates that Ascend NPUs are capable of efficiently and effectively training dense models with more than 100 billion parameters. Our model and system will be available for our commercial customers. |
| title | Pangu Ultra: Pushing the Limits of Dense Large Language Models on Ascend NPUs |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2504.07866 |