Unleashing Low-Bit Inference on Ascend NPUs: A Comprehensive Evaluation of HiFloat Formats
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2026
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866910035629572096 |
|---|---|
| author | Zhao, Pengxiang Zhen, Hui-Ling Li, Xing Bao, Han Lin, Weizhe Yang, Zhiyuan Zhang, Manyi Luo, Yuanyong Yu, Ziwei Wang, Xin Yuan, Mingxuan Yu, Xianzhi Dong, Zhenhua |
| author_facet | Zhao, Pengxiang Zhen, Hui-Ling Li, Xing Bao, Han Lin, Weizhe Yang, Zhiyuan Zhang, Manyi Luo, Yuanyong Yu, Ziwei Wang, Xin Yuan, Mingxuan Yu, Xianzhi Dong, Zhenhua |
| contents | As LLMs scale, low-bit floating-point formats like MXFP and NVFP4 offer new opportunities for precision and efficiency. In this work, we evaluate HiFloat (HiF8 and HiF4), a family of formats tailored for Ascend NPUs. Through rigorous comparison across weight-activation and KV-cache tasks, we provide three key insights: (1) INT8 suits narrow-range data, while floating-point formats excel with high-variance data; (2) in 4-bit regimes, HiF4's hierarchical scaling prevents the accuracy collapse seen in integer formats; and (3) HiFloat is fully compatible with state-of-the-art post-training quantization frameworks. Overall, HiFloat provides a solution for high-efficiency LLM inference on NPUs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_12635 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Unleashing Low-Bit Inference on Ascend NPUs: A Comprehensive Evaluation of HiFloat Formats Zhao, Pengxiang Zhen, Hui-Ling Li, Xing Bao, Han Lin, Weizhe Yang, Zhiyuan Zhang, Manyi Luo, Yuanyong Yu, Ziwei Wang, Xin Yuan, Mingxuan Yu, Xianzhi Dong, Zhenhua Computation and Language Artificial Intelligence Machine Learning As LLMs scale, low-bit floating-point formats like MXFP and NVFP4 offer new opportunities for precision and efficiency. In this work, we evaluate HiFloat (HiF8 and HiF4), a family of formats tailored for Ascend NPUs. Through rigorous comparison across weight-activation and KV-cache tasks, we provide three key insights: (1) INT8 suits narrow-range data, while floating-point formats excel with high-variance data; (2) in 4-bit regimes, HiF4's hierarchical scaling prevents the accuracy collapse seen in integer formats; and (3) HiFloat is fully compatible with state-of-the-art post-training quantization frameworks. Overall, HiFloat provides a solution for high-efficiency LLM inference on NPUs. |
| title | Unleashing Low-Bit Inference on Ascend NPUs: A Comprehensive Evaluation of HiFloat Formats |
| topic | Computation and Language Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2602.12635 |