InDistill: Information flow-preserving knowledge distillation for model compression
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2022
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912198355320832 |
|---|---|
| author | Sarridis, Ioannis Koutlis, Christos Kordopatis-Zilos, Giorgos Kompatsiaris, Ioannis Papadopoulos, Symeon |
| author_facet | Sarridis, Ioannis Koutlis, Christos Kordopatis-Zilos, Giorgos Kompatsiaris, Ioannis Papadopoulos, Symeon |
| contents | In this paper, we introduce InDistill, a method that serves as a warmup stage for enhancing Knowledge Distillation (KD) effectiveness. InDistill focuses on transferring critical information flow paths from a heavyweight teacher to a lightweight student. This is achieved via a training scheme based on curriculum learning that considers the distillation difficulty of each layer and the critical learning periods when the information flow paths are established. This procedure can lead to a student model that is better prepared to learn from the teacher. To ensure the applicability of InDistill across a wide range of teacher-student pairs, we also incorporate a pruning operation when there is a discrepancy in the width of the teacher and student layers. This pruning operation reduces the width of the teacher's intermediate layers to match those of the student, allowing direct distillation without the need for an encoding stage. The proposed method is extensively evaluated using various pairs of teacher-student architectures on CIFAR-10, CIFAR-100, and ImageNet datasets demonstrating that preserving the information flow paths consistently increases the performance of the baseline KD approaches on both classification and retrieval settings. The code is available at https://github.com/gsarridis/InDistill. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2205_10003 |
| institution | arXiv |
| publishDate | 2022 |
| record_format | arxiv |
| spellingShingle | InDistill: Information flow-preserving knowledge distillation for model compression Sarridis, Ioannis Koutlis, Christos Kordopatis-Zilos, Giorgos Kompatsiaris, Ioannis Papadopoulos, Symeon Computer Vision and Pattern Recognition In this paper, we introduce InDistill, a method that serves as a warmup stage for enhancing Knowledge Distillation (KD) effectiveness. InDistill focuses on transferring critical information flow paths from a heavyweight teacher to a lightweight student. This is achieved via a training scheme based on curriculum learning that considers the distillation difficulty of each layer and the critical learning periods when the information flow paths are established. This procedure can lead to a student model that is better prepared to learn from the teacher. To ensure the applicability of InDistill across a wide range of teacher-student pairs, we also incorporate a pruning operation when there is a discrepancy in the width of the teacher and student layers. This pruning operation reduces the width of the teacher's intermediate layers to match those of the student, allowing direct distillation without the need for an encoding stage. The proposed method is extensively evaluated using various pairs of teacher-student architectures on CIFAR-10, CIFAR-100, and ImageNet datasets demonstrating that preserving the information flow paths consistently increases the performance of the baseline KD approaches on both classification and retrieval settings. The code is available at https://github.com/gsarridis/InDistill. |
| title | InDistill: Information flow-preserving knowledge distillation for model compression |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2205.10003 |