HiFloat4 Format for Language Model Pre-training on Ascend NPUs
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2026
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866911580649684992 |
|---|---|
| author | Taghian, Mehran Peng, Yunke Huang, Xing Wang, Yao Wang, Yaoyuan Guo, Wei Luo, Yuanyong Hu, Tianchi Wang, Junsong Wang, Xin Liu, Hu Cheng, Yu Yu, Ziwei Li, Hongliang Rahimifar, Mehdi Yan, Lei Wang, Xuefei Ma, Zhuang Liu, Lei Yu, Hui Raju, Anandharaju Durai Le, Hoang Mak, Hei Yi Rahman, Tanzila Golestan, Shadan |
| author_facet | Taghian, Mehran Peng, Yunke Huang, Xing Wang, Yao Wang, Yaoyuan Guo, Wei Luo, Yuanyong Hu, Tianchi Wang, Junsong Wang, Xin Liu, Hu Cheng, Yu Yu, Ziwei Li, Hongliang Rahimifar, Mehdi Yan, Lei Wang, Xuefei Ma, Zhuang Liu, Lei Yu, Hui Raju, Anandharaju Durai Le, Hoang Mak, Hei Yi Rahman, Tanzila Golestan, Shadan |
| contents | Large foundation models have become central to modern machine learning, with performance scaling predictably with model size and data. However, training and deploying such models incur substantial computational and memory costs, motivating the development of low-precision training techniques. Recent work has demonstrated that 4-bit floating-point (FP4) formats--such as MXFP4 and NVFP4--can be successfully applied to linear GEMM operations in large language models (LLMs), achieving up to 4x improvements in compute throughput and memory efficiency compared to higher-precision baselines. In this work, we investigate the recently proposed HiFloat4 FP4 format for Huawei Ascend NPUs and systematically compare it with MXFP4 in large-scale training settings. All experiments are conducted on Ascend NPU clusters, with linear and expert GEMM operations performed entirely in FP4 precision. We evaluate both dense architectures (e.g., Pangu and LLaMA-style models) and mixture-of-experts (MoE) models, where both standard linear layers and expert-specific GEMMs operate in FP4. Furthermore, we explore stabilization techniques tailored to FP4 training that significantly reduce numerical degradation, maintaining relative error within 1% of full-precision baselines while preserving the efficiency benefits of 4-bit computation. Our results provide a comprehensive empirical study of FP4 training on NPUs and highlight the practical trade-offs between FP4 formats in large-scale dense and MoE models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_08826 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | HiFloat4 Format for Language Model Pre-training on Ascend NPUs Taghian, Mehran Peng, Yunke Huang, Xing Wang, Yao Wang, Yaoyuan Guo, Wei Luo, Yuanyong Hu, Tianchi Wang, Junsong Wang, Xin Liu, Hu Cheng, Yu Yu, Ziwei Li, Hongliang Rahimifar, Mehdi Yan, Lei Wang, Xuefei Ma, Zhuang Liu, Lei Yu, Hui Raju, Anandharaju Durai Le, Hoang Mak, Hei Yi Rahman, Tanzila Golestan, Shadan Machine Learning Artificial Intelligence Computation and Language Large foundation models have become central to modern machine learning, with performance scaling predictably with model size and data. However, training and deploying such models incur substantial computational and memory costs, motivating the development of low-precision training techniques. Recent work has demonstrated that 4-bit floating-point (FP4) formats--such as MXFP4 and NVFP4--can be successfully applied to linear GEMM operations in large language models (LLMs), achieving up to 4x improvements in compute throughput and memory efficiency compared to higher-precision baselines. In this work, we investigate the recently proposed HiFloat4 FP4 format for Huawei Ascend NPUs and systematically compare it with MXFP4 in large-scale training settings. All experiments are conducted on Ascend NPU clusters, with linear and expert GEMM operations performed entirely in FP4 precision. We evaluate both dense architectures (e.g., Pangu and LLaMA-style models) and mixture-of-experts (MoE) models, where both standard linear layers and expert-specific GEMMs operate in FP4. Furthermore, we explore stabilization techniques tailored to FP4 training that significantly reduce numerical degradation, maintaining relative error within 1% of full-precision baselines while preserving the efficiency benefits of 4-bit computation. Our results provide a comprehensive empirical study of FP4 training on NPUs and highlight the practical trade-offs between FP4 formats in large-scale dense and MoE models. |
| title | HiFloat4 Format for Language Model Pre-training on Ascend NPUs |
| topic | Machine Learning Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2604.08826 |