Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2502.11458 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910831312109568 |
|---|---|
| author | Zhou, Jiecheng Tang, Ding Fu, Rong Hu, Boni Xu, Haoran Wang, Yi Pei, Zhilin Su, Zhongling Liu, Liang Zhang, Xingcheng Zhang, Weiming |
| author_facet | Zhou, Jiecheng Tang, Ding Fu, Rong Hu, Boni Xu, Haoran Wang, Yi Pei, Zhilin Su, Zhongling Liu, Liang Zhang, Xingcheng Zhang, Weiming |
| contents | The burgeoning computational demands for training large language models (LLMs) necessitate efficient methods, including quantized training, which leverages low-bit arithmetic operations to reduce costs. While FP8 precision has shown potential, leveraging FP4 remains challenging due to inherent quantization errors and limited representation capability. Based on the Transformer architecture, we present an FP4 training scheme for LLMs, overcoming these obstacles through mixed-precision quantization strategies tailed for different modules and training stages. This allows us to apply the precision level suitable to distinct components within the model, ensuring that multi-head attention and linear layers are handled appropriately. Our pretraining recipe ensures stability in backpropagation by incorporating fine-grained quantization methods with a target precision training schedule. Experimental results demonstrate that our FP4 training scheme achieves accuracy comparable to BF16 and FP8, with smaller theoretical computational cost. With the advent of next-generation hardware supporting FP4, our method sets the foundation for efficient ultra-low precision training. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_11458 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Towards Efficient Pre-training: Exploring FP4 Precision in Large Language Models Zhou, Jiecheng Tang, Ding Fu, Rong Hu, Boni Xu, Haoran Wang, Yi Pei, Zhilin Su, Zhongling Liu, Liang Zhang, Xingcheng Zhang, Weiming Machine Learning Artificial Intelligence I.2 The burgeoning computational demands for training large language models (LLMs) necessitate efficient methods, including quantized training, which leverages low-bit arithmetic operations to reduce costs. While FP8 precision has shown potential, leveraging FP4 remains challenging due to inherent quantization errors and limited representation capability. Based on the Transformer architecture, we present an FP4 training scheme for LLMs, overcoming these obstacles through mixed-precision quantization strategies tailed for different modules and training stages. This allows us to apply the precision level suitable to distinct components within the model, ensuring that multi-head attention and linear layers are handled appropriately. Our pretraining recipe ensures stability in backpropagation by incorporating fine-grained quantization methods with a target precision training schedule. Experimental results demonstrate that our FP4 training scheme achieves accuracy comparable to BF16 and FP8, with smaller theoretical computational cost. With the advent of next-generation hardware supporting FP4, our method sets the foundation for efficient ultra-low precision training. |
| title | Towards Efficient Pre-training: Exploring FP4 Precision in Large Language Models |
| topic | Machine Learning Artificial Intelligence I.2 |
| url | https://arxiv.org/abs/2502.11458 |