Dynamic Contrastive Knowledge Distillation for Efficient Image Restoration
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929634910666752 |
|---|---|
| author | Zhou, Yunshuai Qiao, Junbo Liao, Jincheng Li, Wei Li, Simiao Xie, Jiao Shen, Yunhang Hu, Jie Lin, Shaohui |
| author_facet | Zhou, Yunshuai Qiao, Junbo Liao, Jincheng Li, Wei Li, Simiao Xie, Jiao Shen, Yunhang Hu, Jie Lin, Shaohui |
| contents | Knowledge distillation (KD) is a valuable yet challenging approach that enhances a compact student network by learning from a high-performance but cumbersome teacher model. However, previous KD methods for image restoration overlook the state of the student during the distillation, adopting a fixed solution space that limits the capability of KD. Additionally, relying solely on L1-type loss struggles to leverage the distribution information of images. In this work, we propose a novel dynamic contrastive knowledge distillation (DCKD) framework for image restoration. Specifically, we introduce dynamic contrastive regularization to perceive the student's learning state and dynamically adjust the distilled solution space using contrastive learning. Additionally, we also propose a distribution mapping module to extract and align the pixel-level category distribution of the teacher and student models. Note that the proposed DCKD is a structure-agnostic distillation framework, which can adapt to different backbones and can be combined with methods that optimize upper-bound constraints to further enhance model performance. Extensive experiments demonstrate that DCKD significantly outperforms the state-of-the-art KD methods across various image restoration tasks and backbones. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_08939 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Dynamic Contrastive Knowledge Distillation for Efficient Image Restoration Zhou, Yunshuai Qiao, Junbo Liao, Jincheng Li, Wei Li, Simiao Xie, Jiao Shen, Yunhang Hu, Jie Lin, Shaohui Computer Vision and Pattern Recognition Knowledge distillation (KD) is a valuable yet challenging approach that enhances a compact student network by learning from a high-performance but cumbersome teacher model. However, previous KD methods for image restoration overlook the state of the student during the distillation, adopting a fixed solution space that limits the capability of KD. Additionally, relying solely on L1-type loss struggles to leverage the distribution information of images. In this work, we propose a novel dynamic contrastive knowledge distillation (DCKD) framework for image restoration. Specifically, we introduce dynamic contrastive regularization to perceive the student's learning state and dynamically adjust the distilled solution space using contrastive learning. Additionally, we also propose a distribution mapping module to extract and align the pixel-level category distribution of the teacher and student models. Note that the proposed DCKD is a structure-agnostic distillation framework, which can adapt to different backbones and can be combined with methods that optimize upper-bound constraints to further enhance model performance. Extensive experiments demonstrate that DCKD significantly outperforms the state-of-the-art KD methods across various image restoration tasks and backbones. |
| title | Dynamic Contrastive Knowledge Distillation for Efficient Image Restoration |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2412.08939 |