Quartet: Native FP4 Training Can Be Optimal for Large Language Models
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914256447864832 |
|---|---|
| author | Castro, Roberto L. Panferov, Andrei Tabesh, Soroush Sieberling, Oliver Chen, Jiale Nikdan, Mahdi Ashkboos, Saleh Alistarh, Dan |
| author_facet | Castro, Roberto L. Panferov, Andrei Tabesh, Soroush Sieberling, Oliver Chen, Jiale Nikdan, Mahdi Ashkboos, Saleh Alistarh, Dan |
| contents | Training large language models (LLMs) models directly in low-precision offers a way to address computational costs by improving both throughput and energy efficiency. For those purposes, NVIDIA's recent Blackwell architecture facilitates very low-precision operations using FP4 variants. Yet, current algorithms for training LLMs in FP4 precision face significant accuracy degradation and often rely on mixed-precision fallbacks. In this paper, we investigate hardware-supported FP4 training and introduce a new approach for accurate, end-to-end FP4 training with all the major computations (i.e., linear layers) in low precision. Through extensive evaluations on Llama-type models, we reveal a new low-precision scaling law that quantifies performance trade-offs across bit-widths and training setups. Guided by this investigation, we design an "optimal" technique in terms of accuracy-vs-computation, called Quartet. We implement Quartet using optimized CUDA kernels tailored for Blackwell, demonstrating that fully FP4-based training is a competitive alternative to FP16 half-precision and to FP8 training. Our code is available at https://github.com/IST-DASLab/Quartet. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_14669 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Quartet: Native FP4 Training Can Be Optimal for Large Language Models Castro, Roberto L. Panferov, Andrei Tabesh, Soroush Sieberling, Oliver Chen, Jiale Nikdan, Mahdi Ashkboos, Saleh Alistarh, Dan Machine Learning Training large language models (LLMs) models directly in low-precision offers a way to address computational costs by improving both throughput and energy efficiency. For those purposes, NVIDIA's recent Blackwell architecture facilitates very low-precision operations using FP4 variants. Yet, current algorithms for training LLMs in FP4 precision face significant accuracy degradation and often rely on mixed-precision fallbacks. In this paper, we investigate hardware-supported FP4 training and introduce a new approach for accurate, end-to-end FP4 training with all the major computations (i.e., linear layers) in low precision. Through extensive evaluations on Llama-type models, we reveal a new low-precision scaling law that quantifies performance trade-offs across bit-widths and training setups. Guided by this investigation, we design an "optimal" technique in terms of accuracy-vs-computation, called Quartet. We implement Quartet using optimized CUDA kernels tailored for Blackwell, demonstrating that fully FP4-based training is a competitive alternative to FP16 half-precision and to FP8 training. Our code is available at https://github.com/IST-DASLab/Quartet. |
| title | Quartet: Native FP4 Training Can Be Optimal for Large Language Models |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2505.14669 |