Saved in:
| Main Authors: | , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2508.14004 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909897372729344 |
|---|---|
| author | Salishev, Sergey Akhremchik, Ian |
| author_facet | Salishev, Sergey Akhremchik, Ian |
| contents | Quantized neural networks can be viewed as a chain of noisy channels, where rounding in each layer reduces capacity as bit-width shrinks; the floating-point (FP) checkpoint sets the maximum input rate. We track capacity dynamics as the average bit-width decreases and identify resulting quantization bottlenecks by casting fine-tuning as a smooth, constrained optimization problem. Our approach employs a fully differentiable Straight-Through Estimator (STE) with learnable bit-width, noise scale and clamp bounds, and enforces a target bit-width via an exterior-point penalty; mild metric smoothing (via distillation) stabilizes training. Despite its simplicity, the method attains competitive accuracy down to the extreme W1A1 setting while retaining the efficiency of STE. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_14004 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | GDNSQ: Gradual Differentiable Noise Scale Quantization for Low-bit Neural Networks Salishev, Sergey Akhremchik, Ian Machine Learning Information Theory Numerical Analysis 68T07, 90C26 I.2.6; E.4; G.1.6 Quantized neural networks can be viewed as a chain of noisy channels, where rounding in each layer reduces capacity as bit-width shrinks; the floating-point (FP) checkpoint sets the maximum input rate. We track capacity dynamics as the average bit-width decreases and identify resulting quantization bottlenecks by casting fine-tuning as a smooth, constrained optimization problem. Our approach employs a fully differentiable Straight-Through Estimator (STE) with learnable bit-width, noise scale and clamp bounds, and enforces a target bit-width via an exterior-point penalty; mild metric smoothing (via distillation) stabilizes training. Despite its simplicity, the method attains competitive accuracy down to the extreme W1A1 setting while retaining the efficiency of STE. |
| title | GDNSQ: Gradual Differentiable Noise Scale Quantization for Low-bit Neural Networks |
| topic | Machine Learning Information Theory Numerical Analysis 68T07, 90C26 I.2.6; E.4; G.1.6 |
| url | https://arxiv.org/abs/2508.14004 |