Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2509.20854 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916969885728768 |
|---|---|
| author | Rehman, Abdur Sharif, S M A Rahaman, Md Abdur Rasool, Mohamed Jismy Aashik Kim, Seongwan Lee, Jaeho |
| author_facet | Rehman, Abdur Sharif, S M A Rahaman, Md Abdur Rasool, Mohamed Jismy Aashik Kim, Seongwan Lee, Jaeho |
| contents | Quantization-aware training (QAT) combined with knowledge distillation (KD) is a promising strategy for compressing Artificial Intelligence (AI) models for deployment on resource-constrained hardware. However, existing QAT-KD methods often struggle to balance task-specific (TS) and distillation losses due to heterogeneous gradient magnitudes, especially under low-bit quantization. We propose Game of Regularizer (GoR), a novel learnable regularization method that adaptively balances TS and KD objectives using only two trainable parameters for dynamic loss weighting. GoR reduces conflict between supervision signals, improves convergence, and boosts the performance of small quantized models (SQMs). Experiments on image classification, object detection (OD), and large language model (LLM) compression show that GoR consistently outperforms state-of-the-art QAT-KD methods. On low-power edge devices, it delivers faster inference while maintaining full-precision accuracy. We also introduce QAT-EKD-GoR, an ensemble distillation framework that uses multiple heterogeneous teacher models. Under optimal conditions, the proposed EKD-GoR can outperform full-precision models, providing a robust solution for real-world deployment. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_20854 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Punching Above Precision: Small Quantized Model Distillation with Learnable Regularizer Rehman, Abdur Sharif, S M A Rahaman, Md Abdur Rasool, Mohamed Jismy Aashik Kim, Seongwan Lee, Jaeho Computer Vision and Pattern Recognition Quantization-aware training (QAT) combined with knowledge distillation (KD) is a promising strategy for compressing Artificial Intelligence (AI) models for deployment on resource-constrained hardware. However, existing QAT-KD methods often struggle to balance task-specific (TS) and distillation losses due to heterogeneous gradient magnitudes, especially under low-bit quantization. We propose Game of Regularizer (GoR), a novel learnable regularization method that adaptively balances TS and KD objectives using only two trainable parameters for dynamic loss weighting. GoR reduces conflict between supervision signals, improves convergence, and boosts the performance of small quantized models (SQMs). Experiments on image classification, object detection (OD), and large language model (LLM) compression show that GoR consistently outperforms state-of-the-art QAT-KD methods. On low-power edge devices, it delivers faster inference while maintaining full-precision accuracy. We also introduce QAT-EKD-GoR, an ensemble distillation framework that uses multiple heterogeneous teacher models. Under optimal conditions, the proposed EKD-GoR can outperform full-precision models, providing a robust solution for real-world deployment. |
| title | Punching Above Precision: Small Quantized Model Distillation with Learnable Regularizer |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2509.20854 |