Cover Image

Training Report of TeleChat3-MoE

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Liu, Xinzhang, Wang, Chao, Yang, Zhihao, Jiang, Zhuo, Zhao, Xuncheng, Wang, Haoran, Li, Lei, He, Dongdong, Liu, Luobin, Yuan, Kaizhe, Gao, Han, Wang, Zihan, Yao, Yitong, Xiong, Sishi, Deng, Wenmin, He, Haowei, Yu, Kaidong, Zhao, Yu, Fang, Ruiyu, Jiang, Yuhao, Li, Yingyan, Hu, Xiaohui, Yu, Xi, Li, Jingqi, Liu, Yanwei, Li, Qingli, Shi, Xinyu, Niu, Junhao, Huang, Chengnuo, Xiao, Yao, Wang, Ruiwen, Li, Fengkai, Pu, Luwen, Jia, Kaipeng, Yao, Fubei, Huang, Yuyao, He, Xuewei, Jiang, Zhuoru, Song, Ruiting, Xue, Rui, Xie, Qiyi, Zhang, Jie, Huang, Zilu, Zhang, Zhaoxi, Lu, Zhilong, Zhang, Yanhan, Zhang, Yin, Xue, Yanlei, Yuan, Zhu, Su, Teng, Jiang, Xin, Song, Shuangyong, Li, Yongxiang, Li, Xuelong
Format:	Preprint
Published:	2025
Subjects:	Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

_version_	1866912796987359232
author	Liu, Xinzhang Wang, Chao Yang, Zhihao Jiang, Zhuo Zhao, Xuncheng Wang, Haoran Li, Lei He, Dongdong Liu, Luobin Yuan, Kaizhe Gao, Han Wang, Zihan Yao, Yitong Xiong, Sishi Deng, Wenmin He, Haowei Yu, Kaidong Zhao, Yu Fang, Ruiyu Jiang, Yuhao Li, Yingyan Hu, Xiaohui Yu, Xi Li, Jingqi Liu, Yanwei Li, Qingli Shi, Xinyu Niu, Junhao Huang, Chengnuo Xiao, Yao Wang, Ruiwen Li, Fengkai Pu, Luwen Jia, Kaipeng Yao, Fubei Huang, Yuyao He, Xuewei Jiang, Zhuoru Song, Ruiting Xue, Rui Xie, Qiyi Zhang, Jie Huang, Zilu Zhang, Zhaoxi Lu, Zhilong Zhang, Yanhan Zhang, Yin Xue, Yanlei Yuan, Zhu Su, Teng Jiang, Xin Song, Shuangyong Li, Yongxiang Li, Xuelong
author_facet	Liu, Xinzhang Wang, Chao Yang, Zhihao Jiang, Zhuo Zhao, Xuncheng Wang, Haoran Li, Lei He, Dongdong Liu, Luobin Yuan, Kaizhe Gao, Han Wang, Zihan Yao, Yitong Xiong, Sishi Deng, Wenmin He, Haowei Yu, Kaidong Zhao, Yu Fang, Ruiyu Jiang, Yuhao Li, Yingyan Hu, Xiaohui Yu, Xi Li, Jingqi Liu, Yanwei Li, Qingli Shi, Xinyu Niu, Junhao Huang, Chengnuo Xiao, Yao Wang, Ruiwen Li, Fengkai Pu, Luwen Jia, Kaipeng Yao, Fubei Huang, Yuyao He, Xuewei Jiang, Zhuoru Song, Ruiting Xue, Rui Xie, Qiyi Zhang, Jie Huang, Zilu Zhang, Zhaoxi Lu, Zhilong Zhang, Yanhan Zhang, Yin Xue, Yanlei Yuan, Zhu Su, Teng Jiang, Xin Song, Shuangyong Li, Yongxiang Li, Xuelong
contents	TeleChat3-MoE is the latest series of TeleChat large language models, featuring a Mixture-of-Experts (MoE) architecture with parameter counts ranging from 105 billion to over one trillion,trained end-to-end on Ascend NPU cluster. This technical report mainly presents the underlying training infrastructure that enables reliable and efficient scaling to frontier model sizes. We detail systematic methodologies for operator-level and end-to-end numerical accuracy verification, ensuring consistency across hardware platforms and distributed parallelism strategies. Furthermore, we introduce a suite of performance optimizations, including interleaved pipeline scheduling, attention-aware data scheduling for long-sequence training,hierarchical and overlapped communication for expert parallelism, and DVM-based operator fusion. A systematic parallelization framework, leveraging analytical estimation and integer linear programming, is also proposed to optimize multi-dimensional parallelism configurations. Additionally, we present methodological approaches to cluster-level optimizations, addressing host- and device-bound bottlenecks during large-scale training tasks. These infrastructure advancements yield significant throughput improvements and near-linear scaling on clusters comprising thousands of devices, providing a robust foundation for large-scale language model development on hardware ecosystems.
format	Preprint
id	arxiv_https___arxiv_org_abs_2512_24157
institution	arXiv
publishDate	2025
record_format	arxiv
spellingShingle	Training Report of TeleChat3-MoE Liu, Xinzhang Wang, Chao Yang, Zhihao Jiang, Zhuo Zhao, Xuncheng Wang, Haoran Li, Lei He, Dongdong Liu, Luobin Yuan, Kaizhe Gao, Han Wang, Zihan Yao, Yitong Xiong, Sishi Deng, Wenmin He, Haowei Yu, Kaidong Zhao, Yu Fang, Ruiyu Jiang, Yuhao Li, Yingyan Hu, Xiaohui Yu, Xi Li, Jingqi Liu, Yanwei Li, Qingli Shi, Xinyu Niu, Junhao Huang, Chengnuo Xiao, Yao Wang, Ruiwen Li, Fengkai Pu, Luwen Jia, Kaipeng Yao, Fubei Huang, Yuyao He, Xuewei Jiang, Zhuoru Song, Ruiting Xue, Rui Xie, Qiyi Zhang, Jie Huang, Zilu Zhang, Zhaoxi Lu, Zhilong Zhang, Yanhan Zhang, Yin Xue, Yanlei Yuan, Zhu Su, Teng Jiang, Xin Song, Shuangyong Li, Yongxiang Li, Xuelong Computation and Language TeleChat3-MoE is the latest series of TeleChat large language models, featuring a Mixture-of-Experts (MoE) architecture with parameter counts ranging from 105 billion to over one trillion,trained end-to-end on Ascend NPU cluster. This technical report mainly presents the underlying training infrastructure that enables reliable and efficient scaling to frontier model sizes. We detail systematic methodologies for operator-level and end-to-end numerical accuracy verification, ensuring consistency across hardware platforms and distributed parallelism strategies. Furthermore, we introduce a suite of performance optimizations, including interleaved pipeline scheduling, attention-aware data scheduling for long-sequence training,hierarchical and overlapped communication for expert parallelism, and DVM-based operator fusion. A systematic parallelization framework, leveraging analytical estimation and integer linear programming, is also proposed to optimize multi-dimensional parallelism configurations. Additionally, we present methodological approaches to cluster-level optimizations, addressing host- and device-bound bottlenecks during large-scale training tasks. These infrastructure advancements yield significant throughput improvements and near-linear scaling on clusters comprising thousands of devices, providing a robust foundation for large-scale language model development on hardware ecosystems.
title	Training Report of TeleChat3-MoE
topic	Computation and Language
url	https://arxiv.org/abs/2512.24157