Fisher-aware Quantization for DETR Detectors with Critical-category Objectives
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917712925556736 |
|---|---|
| author | Yang, Huanrui Huang, Yafeng Dong, Zhen Gudovskiy, Denis A Okuno, Tomoyuki Nakata, Yohei Du, Yuan Keutzer, Kurt Zhang, Shanghang |
| author_facet | Yang, Huanrui Huang, Yafeng Dong, Zhen Gudovskiy, Denis A Okuno, Tomoyuki Nakata, Yohei Du, Yuan Keutzer, Kurt Zhang, Shanghang |
| contents | The impact of quantization on the overall performance of deep learning models is a well-studied problem. However, understanding and mitigating its effects on a more fine-grained level is still lacking, especially for harder tasks such as object detection with both classification and regression objectives. This work defines the performance for a subset of task-critical categories, i.e. the critical-category performance, as a crucial yet largely overlooked fine-grained objective for detection tasks. We analyze the impact of quantization at the category-level granularity, and propose methods to improve performance for the critical categories. Specifically, we find that certain critical categories have a higher sensitivity to quantization, and are prone to overfitting after quantization-aware training (QAT). To explain this, we provide theoretical and empirical links between their performance gaps and the corresponding loss landscapes with the Fisher information framework. Using this evidence, we apply a Fisher-aware mixed-precision quantization scheme, and a Fisher-trace regularization for the QAT on the critical-category loss landscape. The proposed methods improve critical-category metrics of the quantized transformer-based DETR detectors. They are even more significant in case of larger models and higher number of classes where the overfitting becomes more severe. For example, our methods lead to 10.4% and 14.5% mAP gains for, correspondingly, 4-bit DETR-R50 and Deformable DETR on the most impacted critical classes in the COCO Panoptic dataset. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_03442 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Fisher-aware Quantization for DETR Detectors with Critical-category Objectives Yang, Huanrui Huang, Yafeng Dong, Zhen Gudovskiy, Denis A Okuno, Tomoyuki Nakata, Yohei Du, Yuan Keutzer, Kurt Zhang, Shanghang Computer Vision and Pattern Recognition The impact of quantization on the overall performance of deep learning models is a well-studied problem. However, understanding and mitigating its effects on a more fine-grained level is still lacking, especially for harder tasks such as object detection with both classification and regression objectives. This work defines the performance for a subset of task-critical categories, i.e. the critical-category performance, as a crucial yet largely overlooked fine-grained objective for detection tasks. We analyze the impact of quantization at the category-level granularity, and propose methods to improve performance for the critical categories. Specifically, we find that certain critical categories have a higher sensitivity to quantization, and are prone to overfitting after quantization-aware training (QAT). To explain this, we provide theoretical and empirical links between their performance gaps and the corresponding loss landscapes with the Fisher information framework. Using this evidence, we apply a Fisher-aware mixed-precision quantization scheme, and a Fisher-trace regularization for the QAT on the critical-category loss landscape. The proposed methods improve critical-category metrics of the quantized transformer-based DETR detectors. They are even more significant in case of larger models and higher number of classes where the overfitting becomes more severe. For example, our methods lead to 10.4% and 14.5% mAP gains for, correspondingly, 4-bit DETR-R50 and Deformable DETR on the most impacted critical classes in the COCO Panoptic dataset. |
| title | Fisher-aware Quantization for DETR Detectors with Critical-category Objectives |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2407.03442 |