Optimizing Large Language Model Training Using FP4 Quantization
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866917497475694592 |
|---|---|
| author | Wang, Ruizhe Gong, Yeyun Liu, Xiao Zhao, Guoshuai Yang, Ziyue Guo, Baining Zha, Zhengjun Cheng, Peng |
| author_facet | Wang, Ruizhe Gong, Yeyun Liu, Xiao Zhao, Guoshuai Yang, Ziyue Guo, Baining Zha, Zhengjun Cheng, Peng |
| contents | The growing computational demands of training large language models (LLMs) necessitate more efficient methods. Quantized training presents a promising solution by enabling low-bit arithmetic operations to reduce these costs. While FP8 precision has demonstrated feasibility, leveraging FP4 remains a challenge due to significant quantization errors and limited representational capacity. This work introduces the first FP4 training framework for LLMs, addressing these challenges with two key innovations: a differentiable quantization estimator for precise weight updates and an outlier clamping and compensation strategy to prevent activation collapse. To ensure stability, the framework integrates a mixed-precision training scheme and vector-wise quantization. Experimental results demonstrate that our FP4 framework achieves accuracy comparable to BF16 and FP8, with minimal degradation, scaling effectively to 13B-parameter LLMs trained on up to 100B tokens. With the emergence of next-generation hardware supporting FP4, our framework sets a foundation for efficient ultra-low precision training. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_17116 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Optimizing Large Language Model Training Using FP4 Quantization Wang, Ruizhe Gong, Yeyun Liu, Xiao Zhao, Guoshuai Yang, Ziyue Guo, Baining Zha, Zhengjun Cheng, Peng Machine Learning Computation and Language The growing computational demands of training large language models (LLMs) necessitate more efficient methods. Quantized training presents a promising solution by enabling low-bit arithmetic operations to reduce these costs. While FP8 precision has demonstrated feasibility, leveraging FP4 remains a challenge due to significant quantization errors and limited representational capacity. This work introduces the first FP4 training framework for LLMs, addressing these challenges with two key innovations: a differentiable quantization estimator for precise weight updates and an outlier clamping and compensation strategy to prevent activation collapse. To ensure stability, the framework integrates a mixed-precision training scheme and vector-wise quantization. Experimental results demonstrate that our FP4 framework achieves accuracy comparable to BF16 and FP8, with minimal degradation, scaling effectively to 13B-parameter LLMs trained on up to 100B tokens. With the emergence of next-generation hardware supporting FP4, our framework sets a foundation for efficient ultra-low precision training. |
| title | Optimizing Large Language Model Training Using FP4 Quantization |
| topic | Machine Learning Computation and Language |
| url | https://arxiv.org/abs/2501.17116 |