Data Upcycling Knowledge Distillation for Image Super-Resolution
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929329135419392 |
|---|---|
| author | Zhang, Yun Li, Wei Li, Simiao Chen, Hanting Tu, Zhijun Wang, Wenjia Jing, Bingyi Lin, Shaohui Hu, Jie |
| author_facet | Zhang, Yun Li, Wei Li, Simiao Chen, Hanting Tu, Zhijun Wang, Wenjia Jing, Bingyi Lin, Shaohui Hu, Jie |
| contents | Knowledge distillation (KD) compresses deep neural networks by transferring task-related knowledge from cumbersome pre-trained teacher models to compact student models. However, current KD methods for super-resolution (SR) networks overlook the nature of SR task that the outputs of the teacher model are noisy approximations to the ground-truth distribution of high-quality images (GT), which shades the teacher model's knowledge to result in limited KD effects. To utilize the teacher model beyond the GT upper-bound, we present the Data Upcycling Knowledge Distillation (DUKD), to transfer the teacher model's knowledge to the student model through the upcycled in-domain data derived from training data. Besides, we impose label consistency regularization to KD for SR by the paired invertible augmentations to improve the student model's performance and robustness. Comprehensive experiments demonstrate that the DUKD method significantly outperforms previous arts on several SR tasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2309_14162 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Data Upcycling Knowledge Distillation for Image Super-Resolution Zhang, Yun Li, Wei Li, Simiao Chen, Hanting Tu, Zhijun Wang, Wenjia Jing, Bingyi Lin, Shaohui Hu, Jie Computer Vision and Pattern Recognition Artificial Intelligence Knowledge distillation (KD) compresses deep neural networks by transferring task-related knowledge from cumbersome pre-trained teacher models to compact student models. However, current KD methods for super-resolution (SR) networks overlook the nature of SR task that the outputs of the teacher model are noisy approximations to the ground-truth distribution of high-quality images (GT), which shades the teacher model's knowledge to result in limited KD effects. To utilize the teacher model beyond the GT upper-bound, we present the Data Upcycling Knowledge Distillation (DUKD), to transfer the teacher model's knowledge to the student model through the upcycled in-domain data derived from training data. Besides, we impose label consistency regularization to KD for SR by the paired invertible augmentations to improve the student model's performance and robustness. Comprehensive experiments demonstrate that the DUKD method significantly outperforms previous arts on several SR tasks. |
| title | Data Upcycling Knowledge Distillation for Image Super-Resolution |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence |
| url | https://arxiv.org/abs/2309.14162 |