StableQAT: Stable Quantization-Aware Training at Ultra-Low Bitwidths
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2026
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866912911072428032 |
|---|---|
| author | Chen, Tianyi Chen, Sihan Qu, Xiaoyi Zhao, Dan Yan, Ruomei Ko, Jongwoo Liang, Luming Cameron, Pashmina |
| author_facet | Chen, Tianyi Chen, Sihan Qu, Xiaoyi Zhao, Dan Yan, Ruomei Ko, Jongwoo Liang, Luming Cameron, Pashmina |
| contents | Quantization-aware training (QAT) is essential for deploying large models under strict memory and latency constraints, yet achieving stable and robust optimization at ultra-low bitwidths remains challenging. Common approaches based on the straight-through estimator (STE) or soft quantizers often suffer from gradient mismatch, instability, or high computational overhead. As such, we propose StableQAT, a unified and efficient QAT framework that stabilizes training in ultra low-bit settings via a novel, lightweight, and theoretically grounded surrogate for backpropagation derived from a discrete Fourier analysis of the rounding operator. StableQAT strictly generalizes STE as the latter arises as a special case of our more expressive surrogate family, yielding smooth, bounded, and inexpensive gradients that improve QAT training performance and stability across various hyperparameter choices. In experiments, StableQAT exhibits stable and efficient QAT at 2-4 bit regimes, demonstrating improved training stability, robustness, and superior performance with negligible training overhead against standard QAT techniques. Our code is available at https://github.com/microsoft/StableQAT. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_19320 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | StableQAT: Stable Quantization-Aware Training at Ultra-Low Bitwidths Chen, Tianyi Chen, Sihan Qu, Xiaoyi Zhao, Dan Yan, Ruomei Ko, Jongwoo Liang, Luming Cameron, Pashmina Machine Learning Artificial Intelligence Quantization-aware training (QAT) is essential for deploying large models under strict memory and latency constraints, yet achieving stable and robust optimization at ultra-low bitwidths remains challenging. Common approaches based on the straight-through estimator (STE) or soft quantizers often suffer from gradient mismatch, instability, or high computational overhead. As such, we propose StableQAT, a unified and efficient QAT framework that stabilizes training in ultra low-bit settings via a novel, lightweight, and theoretically grounded surrogate for backpropagation derived from a discrete Fourier analysis of the rounding operator. StableQAT strictly generalizes STE as the latter arises as a special case of our more expressive surrogate family, yielding smooth, bounded, and inexpensive gradients that improve QAT training performance and stability across various hyperparameter choices. In experiments, StableQAT exhibits stable and efficient QAT at 2-4 bit regimes, demonstrating improved training stability, robustness, and superior performance with negligible training overhead against standard QAT techniques. Our code is available at https://github.com/microsoft/StableQAT. |
| title | StableQAT: Stable Quantization-Aware Training at Ultra-Low Bitwidths |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2601.19320 |