Metis: Training LLMs with FP4 Quantization
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916978646581248 |
|---|---|
| author | Cao, Hengjie Chen, Mengyi Yang, Yifeng Huang, Ruijun Dong, Fang Zhou, Jixian Chen, Anrui Dong, Mingzhi Wang, Yujiang Hou, Jinlong Cheng, Yuan Wu, Fan Yang, Fan Lu, Tun Gu, Ning Shang, Li |
| author_facet | Cao, Hengjie Chen, Mengyi Yang, Yifeng Huang, Ruijun Dong, Fang Zhou, Jixian Chen, Anrui Dong, Mingzhi Wang, Yujiang Hou, Jinlong Cheng, Yuan Wu, Fan Yang, Fan Lu, Tun Gu, Ning Shang, Li |
| contents | This work identifies anisotropy in the singular value spectra of parameters, activations, and gradients as the fundamental barrier to low-bit training of large language models (LLMs). These spectra are dominated by a small fraction of large singular values, inducing wide numerical ranges that cause quantization bias and severe spectral distortion, ultimately degrading training performance. This work presents Metis, a spectral-domain quantization framework that partitions anisotropic spectra into narrower sub-distributions for independent quantization, thereby reducing errors and preserving spectral structure. To minimize overhead, Metis leverages two key properties of the dominant spectral subspace: preservation via sparsely random sampling and preservation via random projection, reducing decomposition cost to a negligible level. On LLaMA-3 8B trained with 100B tokens, Metis enables robust W4A4G4 training with FP4 quantization of weights, activations, and gradients, yielding only a 0.4% training loss gap and a 0.1% degradation in downstream accuracy relative to BF16. Beyond matching BF16 fidelity, Metis also surpasses our implementation of Nvidia's recently announced (yet to be publicly released) FP4 recipe, consistently achieving lower loss and higher downstream accuracy while incurring significantly lower computational overhead. The code implementation for Metis is available at: https://anonymous.4open.science/r/Metis-quantization-644B. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_00404 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Metis: Training LLMs with FP4 Quantization Cao, Hengjie Chen, Mengyi Yang, Yifeng Huang, Ruijun Dong, Fang Zhou, Jixian Chen, Anrui Dong, Mingzhi Wang, Yujiang Hou, Jinlong Cheng, Yuan Wu, Fan Yang, Fan Lu, Tun Gu, Ning Shang, Li Machine Learning This work identifies anisotropy in the singular value spectra of parameters, activations, and gradients as the fundamental barrier to low-bit training of large language models (LLMs). These spectra are dominated by a small fraction of large singular values, inducing wide numerical ranges that cause quantization bias and severe spectral distortion, ultimately degrading training performance. This work presents Metis, a spectral-domain quantization framework that partitions anisotropic spectra into narrower sub-distributions for independent quantization, thereby reducing errors and preserving spectral structure. To minimize overhead, Metis leverages two key properties of the dominant spectral subspace: preservation via sparsely random sampling and preservation via random projection, reducing decomposition cost to a negligible level. On LLaMA-3 8B trained with 100B tokens, Metis enables robust W4A4G4 training with FP4 quantization of weights, activations, and gradients, yielding only a 0.4% training loss gap and a 0.1% degradation in downstream accuracy relative to BF16. Beyond matching BF16 fidelity, Metis also surpasses our implementation of Nvidia's recently announced (yet to be publicly released) FP4 recipe, consistently achieving lower loss and higher downstream accuracy while incurring significantly lower computational overhead. The code implementation for Metis is available at: https://anonymous.4open.science/r/Metis-quantization-644B. |
| title | Metis: Training LLMs with FP4 Quantization |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2509.00404 |