QuantNAS for super resolution: searching for efficient quantization-friendly architectures against quantization noise
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2022
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916086329376768 |
|---|---|
| author | Shvetsov, Egor Osin, Dmitry Zaytsev, Alexey Koryakovskiy, Ivan Buchnev, Valentin Trofimov, Ilya Burnaev, Evgeny |
| author_facet | Shvetsov, Egor Osin, Dmitry Zaytsev, Alexey Koryakovskiy, Ivan Buchnev, Valentin Trofimov, Ilya Burnaev, Evgeny |
| contents | There is a constant need for high-performing and computationally efficient neural network models for image super-resolution: computationally efficient models can be used via low-capacity devices and reduce carbon footprints. One way to obtain such models is to compress models, e.g. quantization. Another way is a neural architecture search that automatically discovers new, more efficient solutions. We propose a novel quantization-aware procedure, the QuantNAS that combines pros of these two approaches. To make QuantNAS work, the procedure looks for quantization-friendly super-resolution models. The approach utilizes entropy regularization, quantization noise, and Adaptive Deviation for Quantization (ADQ) module to enhance the search procedure. The entropy regularization technique prioritizes a single operation within each block of the search space. Adding quantization noise to parameters and activations approximates model degradation after quantization, resulting in a more quantization-friendly architectures. ADQ helps to alleviate problems caused by Batch Norm blocks in super-resolution models. Our experimental results show that the proposed approximations are better for search procedure than direct model quantization. QuantNAS discovers architectures with better PSNR/BitOps trade-off than uniform or mixed precision quantization of fixed architectures. We showcase the effectiveness of our method through its application to two search spaces inspired by the state-of-the-art SR models and RFDN. Thus, anyone can design a proper search space based on an existing architecture and apply our method to obtain better quality and efficiency.
The proposed procedure is 30\% faster than direct weight quantization and is more stable. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2208_14839 |
| institution | arXiv |
| publishDate | 2022 |
| record_format | arxiv |
| spellingShingle | QuantNAS for super resolution: searching for efficient quantization-friendly architectures against quantization noise Shvetsov, Egor Osin, Dmitry Zaytsev, Alexey Koryakovskiy, Ivan Buchnev, Valentin Trofimov, Ilya Burnaev, Evgeny Computer Vision and Pattern Recognition There is a constant need for high-performing and computationally efficient neural network models for image super-resolution: computationally efficient models can be used via low-capacity devices and reduce carbon footprints. One way to obtain such models is to compress models, e.g. quantization. Another way is a neural architecture search that automatically discovers new, more efficient solutions. We propose a novel quantization-aware procedure, the QuantNAS that combines pros of these two approaches. To make QuantNAS work, the procedure looks for quantization-friendly super-resolution models. The approach utilizes entropy regularization, quantization noise, and Adaptive Deviation for Quantization (ADQ) module to enhance the search procedure. The entropy regularization technique prioritizes a single operation within each block of the search space. Adding quantization noise to parameters and activations approximates model degradation after quantization, resulting in a more quantization-friendly architectures. ADQ helps to alleviate problems caused by Batch Norm blocks in super-resolution models. Our experimental results show that the proposed approximations are better for search procedure than direct model quantization. QuantNAS discovers architectures with better PSNR/BitOps trade-off than uniform or mixed precision quantization of fixed architectures. We showcase the effectiveness of our method through its application to two search spaces inspired by the state-of-the-art SR models and RFDN. Thus, anyone can design a proper search space based on an existing architecture and apply our method to obtain better quality and efficiency. The proposed procedure is 30\% faster than direct weight quantization and is more stable. |
| title | QuantNAS for super resolution: searching for efficient quantization-friendly architectures against quantization noise |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2208.14839 |