MixFP4: Enhancing NVFP4 with Adaptive FP4/INT4 Block Representations
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914616246796288 |
|---|---|
| author | Zou, Jiaxiang Chen, Yonghao Wu, Ruilong Chen, Xinyu |
| author_facet | Zou, Jiaxiang Chen, Yonghao Wu, Ruilong Chen, Xinyu |
| contents | As large language models continue to scale, fine-grained block-scaled low-precision formats such as NVFP4 are increasingly adopted for their substantial throughput and memory benefits. However, a single FP4 micro-format often mismatches heterogeneous block-level tensor statistics. To address this without changing the standard block-scaled MMA/GEMM execution path, we propose MixFP4, a mixed micro-format extension to NVFP4 that selects between two stored FP4 micro-formats (E2M1 and E1M2) per block. MixFP4 reuses NVFP4's scale hierarchy and encodes the format choice with zero additional metadata by repurposing the sign bit of the FP8 E4M3 block scale. By decoding both micro-formats into a unified internal E2M2 compute representation, MixFP4 avoids datapath duplication. Across representative LLM families, MixFP4 improves FP4 quantization robustness and accuracy over NVFP4 baselines with modest tensor-core overhead (3.1\% area, 1.5\% power). |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_31035 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | MixFP4: Enhancing NVFP4 with Adaptive FP4/INT4 Block Representations Zou, Jiaxiang Chen, Yonghao Wu, Ruilong Chen, Xinyu Hardware Architecture As large language models continue to scale, fine-grained block-scaled low-precision formats such as NVFP4 are increasingly adopted for their substantial throughput and memory benefits. However, a single FP4 micro-format often mismatches heterogeneous block-level tensor statistics. To address this without changing the standard block-scaled MMA/GEMM execution path, we propose MixFP4, a mixed micro-format extension to NVFP4 that selects between two stored FP4 micro-formats (E2M1 and E1M2) per block. MixFP4 reuses NVFP4's scale hierarchy and encodes the format choice with zero additional metadata by repurposing the sign bit of the FP8 E4M3 block scale. By decoding both micro-formats into a unified internal E2M2 compute representation, MixFP4 avoids datapath duplication. Across representative LLM families, MixFP4 improves FP4 quantization robustness and accuracy over NVFP4 baselines with modest tensor-core overhead (3.1\% area, 1.5\% power). |
| title | MixFP4: Enhancing NVFP4 with Adaptive FP4/INT4 Block Representations |
| topic | Hardware Architecture |
| url | https://arxiv.org/abs/2605.31035 |