High-Dimensional Learning Dynamics of Quantized Models with Straight-Through Estimator
Fuente:
arXiv
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915548907962368 |
|---|---|
| author | Ichikawa, Yuma Kashiwamura, Shuhei Sakata, Ayaka |
| author_facet | Ichikawa, Yuma Kashiwamura, Shuhei Sakata, Ayaka |
| contents | Quantized neural network training optimizes a discrete, non-differentiable objective. The straight-through estimator (STE) enables backpropagation through surrogate gradients and is widely used. While previous studies have primarily focused on the properties of surrogate gradients and their convergence, the influence of quantization hyperparameters, such as bit width and quantization range, on learning dynamics remains largely unexplored. We theoretically show that in the high-dimensional limit, STE dynamics converge to a deterministic ordinary differential equation. This reveals that STE training exhibits a plateau followed by a sharp drop in generalization error, with plateau length depending on the quantization range. A fixed-point analysis quantifies the asymptotic deviation from the unquantized linear model. We also extend analytical techniques for stochastic gradient descent to nonlinear transformations of weights and inputs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_10693 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | High-Dimensional Learning Dynamics of Quantized Models with Straight-Through Estimator Ichikawa, Yuma Kashiwamura, Shuhei Sakata, Ayaka Machine Learning Disordered Systems and Neural Networks Artificial Intelligence Statistics Theory Quantized neural network training optimizes a discrete, non-differentiable objective. The straight-through estimator (STE) enables backpropagation through surrogate gradients and is widely used. While previous studies have primarily focused on the properties of surrogate gradients and their convergence, the influence of quantization hyperparameters, such as bit width and quantization range, on learning dynamics remains largely unexplored. We theoretically show that in the high-dimensional limit, STE dynamics converge to a deterministic ordinary differential equation. This reveals that STE training exhibits a plateau followed by a sharp drop in generalization error, with plateau length depending on the quantization range. A fixed-point analysis quantifies the asymptotic deviation from the unquantized linear model. We also extend analytical techniques for stochastic gradient descent to nonlinear transformations of weights and inputs. |
| title | High-Dimensional Learning Dynamics of Quantized Models with Straight-Through Estimator |
| topic | Machine Learning Disordered Systems and Neural Networks Artificial Intelligence Statistics Theory |
| url | https://arxiv.org/abs/2510.10693 |