Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2404.11769 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910415027437568 |
|---|---|
| author | AskariHemmat, MohammadHossein Jeddi, Ahmadreza Hemmat, Reyhane Askari Lazarevich, Ivan Hoffman, Alexander Sah, Sudhakar Saboori, Ehsan Savaria, Yvon David, Jean-Pierre |
| author_facet | AskariHemmat, MohammadHossein Jeddi, Ahmadreza Hemmat, Reyhane Askari Lazarevich, Ivan Hoffman, Alexander Sah, Sudhakar Saboori, Ehsan Savaria, Yvon David, Jean-Pierre |
| contents | Quantization lowers memory usage, computational requirements, and latency by utilizing fewer bits to represent model weights and activations. In this work, we investigate the generalization properties of quantized neural networks, a characteristic that has received little attention despite its implications on model performance. In particular, first, we develop a theoretical model for quantization in neural networks and demonstrate how quantization functions as a form of regularization. Second, motivated by recent work connecting the sharpness of the loss landscape and generalization, we derive an approximate bound for the generalization of quantized models conditioned on the amount of quantization noise. We then validate our hypothesis by experimenting with over 2000 models trained on CIFAR-10, CIFAR-100, and ImageNet datasets on convolutional and transformer-based models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_11769 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | QGen: On the Ability to Generalize in Quantization Aware Training AskariHemmat, MohammadHossein Jeddi, Ahmadreza Hemmat, Reyhane Askari Lazarevich, Ivan Hoffman, Alexander Sah, Sudhakar Saboori, Ehsan Savaria, Yvon David, Jean-Pierre Machine Learning Computer Vision and Pattern Recognition Quantization lowers memory usage, computational requirements, and latency by utilizing fewer bits to represent model weights and activations. In this work, we investigate the generalization properties of quantized neural networks, a characteristic that has received little attention despite its implications on model performance. In particular, first, we develop a theoretical model for quantization in neural networks and demonstrate how quantization functions as a form of regularization. Second, motivated by recent work connecting the sharpness of the loss landscape and generalization, we derive an approximate bound for the generalization of quantized models conditioned on the amount of quantization noise. We then validate our hypothesis by experimenting with over 2000 models trained on CIFAR-10, CIFAR-100, and ImageNet datasets on convolutional and transformer-based models. |
| title | QGen: On the Ability to Generalize in Quantization Aware Training |
| topic | Machine Learning Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2404.11769 |