Rotation Invariant Quantization for Model Compression
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915040951533568 |
|---|---|
| author | Kampeas, Joseph Nahshan, Yury Kremer, Hanoch Lederman, Gil Zaloshinski, Shira Li, Zheng Haleva, Emir |
| author_facet | Kampeas, Joseph Nahshan, Yury Kremer, Hanoch Lederman, Gil Zaloshinski, Shira Li, Zheng Haleva, Emir |
| contents | Post-training Neural Network (NN) model compression is an attractive approach for deploying large, memory-consuming models on devices with limited memory resources. In this study, we investigate the rate-distortion tradeoff for NN model compression. First, we suggest a Rotation-Invariant Quantization (RIQ) technique that utilizes a single parameter to quantize the entire NN model, yielding a different rate at each layer, i.e., mixed-precision quantization. Then, we prove that our rotation-invariant approach is optimal in terms of compression. We rigorously evaluate RIQ and demonstrate its capabilities on various models and tasks. For example, RIQ facilitates $\times 19.4$ and $\times 52.9$ compression ratios on pre-trained VGG dense and pruned models, respectively, with $<0.4\%$ accuracy degradation. Code is available in \href{https://github.com/ehaleva/RIQ}{github.com/ehaleva/RIQ}. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2303_03106 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Rotation Invariant Quantization for Model Compression Kampeas, Joseph Nahshan, Yury Kremer, Hanoch Lederman, Gil Zaloshinski, Shira Li, Zheng Haleva, Emir Machine Learning Artificial Intelligence Information Theory I.2.4; E.4 Post-training Neural Network (NN) model compression is an attractive approach for deploying large, memory-consuming models on devices with limited memory resources. In this study, we investigate the rate-distortion tradeoff for NN model compression. First, we suggest a Rotation-Invariant Quantization (RIQ) technique that utilizes a single parameter to quantize the entire NN model, yielding a different rate at each layer, i.e., mixed-precision quantization. Then, we prove that our rotation-invariant approach is optimal in terms of compression. We rigorously evaluate RIQ and demonstrate its capabilities on various models and tasks. For example, RIQ facilitates $\times 19.4$ and $\times 52.9$ compression ratios on pre-trained VGG dense and pruned models, respectively, with $<0.4\%$ accuracy degradation. Code is available in \href{https://github.com/ehaleva/RIQ}{github.com/ehaleva/RIQ}. |
| title | Rotation Invariant Quantization for Model Compression |
| topic | Machine Learning Artificial Intelligence Information Theory I.2.4; E.4 |
| url | https://arxiv.org/abs/2303.03106 |