Four Over Six: More Accurate NVFP4 Quantization with Adaptive Block Scaling
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911665827610624 |
|---|---|
| author | Cook, Jack Guo, Junxian Xiao, Guangxuan Lin, Yujun Wyss, Keith Nazemi, Mahdi Mishra, Asit del Mundo, Carlo Blankevoort, Tijmen Han, Song |
| author_facet | Cook, Jack Guo, Junxian Xiao, Guangxuan Lin, Yujun Wyss, Keith Nazemi, Mahdi Mishra, Asit del Mundo, Carlo Blankevoort, Tijmen Han, Song |
| contents | As large language models have grown larger, interest has grown in low-precision numerical formats such as NVFP4 as a way to improve speed and reduce memory usage. However, quantizing models to NVFP4 remains challenging as the lack of precision generally degrades model performance. In this work, we address this issue with Four Over Six (4/6), a modification to the block-scaled NVFP4 quantization algorithm that yields reduced quantization error. Unlike integer formats, floating point formats have non-uniform step sizes which create larger quantization error on larger values. 4/6 takes advantage of this by adaptively scaling some blocks to smaller FP4 values, making the distribution of representable values more uniform and reducing quantization error for near-maximal values. We show that 4/6 can be implemented efficiently on modern hardware accelerators, resulting in performance gains during both pre-training and inference with minimal computational overhead. In pre-training experiments with the Nemotron 3 Nano 30B-A3B model architecture, we find that 4/6 brings training loss closer to BF16 compared to models trained with current state-of-the-art NVFP4 training recipes. Our code is available at https://github.com/mit-han-lab/fouroversix. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_02010 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Four Over Six: More Accurate NVFP4 Quantization with Adaptive Block Scaling Cook, Jack Guo, Junxian Xiao, Guangxuan Lin, Yujun Wyss, Keith Nazemi, Mahdi Mishra, Asit del Mundo, Carlo Blankevoort, Tijmen Han, Song Computation and Language Machine Learning As large language models have grown larger, interest has grown in low-precision numerical formats such as NVFP4 as a way to improve speed and reduce memory usage. However, quantizing models to NVFP4 remains challenging as the lack of precision generally degrades model performance. In this work, we address this issue with Four Over Six (4/6), a modification to the block-scaled NVFP4 quantization algorithm that yields reduced quantization error. Unlike integer formats, floating point formats have non-uniform step sizes which create larger quantization error on larger values. 4/6 takes advantage of this by adaptively scaling some blocks to smaller FP4 values, making the distribution of representable values more uniform and reducing quantization error for near-maximal values. We show that 4/6 can be implemented efficiently on modern hardware accelerators, resulting in performance gains during both pre-training and inference with minimal computational overhead. In pre-training experiments with the Nemotron 3 Nano 30B-A3B model architecture, we find that 4/6 brings training loss closer to BF16 compared to models trained with current state-of-the-art NVFP4 training recipes. Our code is available at https://github.com/mit-han-lab/fouroversix. |
| title | Four Over Six: More Accurate NVFP4 Quantization with Adaptive Block Scaling |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2512.02010 |