Latent-Space Mean-Field Theory for Deep BitNet-like Training: Constrained Gradient Flows with Smooth Quantization and STE Limits
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866908510432788480 |
|---|---|
| author | Kim, Dongwon Lee, Dongseok |
| author_facet | Kim, Dongwon Lee, Dongseok |
| contents | This work develops a mean-field analysis for the asymptotic behavior of deep BitNet-like architectures as smooth quantization parameters approach zero. We establish that empirical measures of latent weights converge weakly to solutions of constrained continuity equations under vanishing quantization smoothing. Our main theoretical contribution demonstrates that the natural exponential decay in smooth quantization cancels out apparent singularities, yielding uniform bounds on mean-field dynamics independent of smoothing parameters. Under standard regularity assumptions, we prove convergence to a well-defined limit that provides the mathematical foundation for gradient-based training of quantized neural networks through distributional analysis. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_00133 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Latent-Space Mean-Field Theory for Deep BitNet-like Training: Constrained Gradient Flows with Smooth Quantization and STE Limits Kim, Dongwon Lee, Dongseok Optimization and Control This work develops a mean-field analysis for the asymptotic behavior of deep BitNet-like architectures as smooth quantization parameters approach zero. We establish that empirical measures of latent weights converge weakly to solutions of constrained continuity equations under vanishing quantization smoothing. Our main theoretical contribution demonstrates that the natural exponential decay in smooth quantization cancels out apparent singularities, yielding uniform bounds on mean-field dynamics independent of smoothing parameters. Under standard regularity assumptions, we prove convergence to a well-defined limit that provides the mathematical foundation for gradient-based training of quantized neural networks through distributional analysis. |
| title | Latent-Space Mean-Field Theory for Deep BitNet-like Training: Constrained Gradient Flows with Smooth Quantization and STE Limits |
| topic | Optimization and Control |
| url | https://arxiv.org/abs/2509.00133 |