Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2310.03669 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918114672771072 |
|---|---|
| author | Hossain, Md. Ismail Elahi, M M Lutfe Ramasinghe, Sameera Cheraghian, Ali Rahman, Fuad Mohammed, Nabeel Rahman, Shafin |
| author_facet | Hossain, Md. Ismail Elahi, M M Lutfe Ramasinghe, Sameera Cheraghian, Ali Rahman, Fuad Mohammed, Nabeel Rahman, Shafin |
| contents | In the knowledge distillation literature, feature-based methods have dominated due to their ability to effectively tap into extensive teacher models. In contrast, logit-based approaches, which aim to distill "dark knowledge" from teachers, typically exhibit inferior performance compared to feature-based methods. To bridge this gap, we present LumiNet, a novel knowledge distillation algorithm designed to enhance logit-based distillation. We introduce the concept of "perception", aiming to calibrate logits based on the model's representation capability. This concept addresses overconfidence issues in the logit-based distillation method while also introducing a novel method to distill knowledge from the teacher. It reconstructs the logits of a sample/instances by considering relationships with other samples in the batch. LumiNet excels on benchmarks like CIFAR-100, ImageNet, and MSCOCO, outperforming the leading feature-based methods, e.g., compared to KD with ResNet18 and MobileNetV2 on ImageNet, it shows improvements of 1.5% and 2.05%, respectively. Codes are available at https://github.com/ismail31416/LumiNet. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2310_03669 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | LumiNet: Perception-Driven Knowledge Distillation via Statistical Logit Calibration Hossain, Md. Ismail Elahi, M M Lutfe Ramasinghe, Sameera Cheraghian, Ali Rahman, Fuad Mohammed, Nabeel Rahman, Shafin Computer Vision and Pattern Recognition In the knowledge distillation literature, feature-based methods have dominated due to their ability to effectively tap into extensive teacher models. In contrast, logit-based approaches, which aim to distill "dark knowledge" from teachers, typically exhibit inferior performance compared to feature-based methods. To bridge this gap, we present LumiNet, a novel knowledge distillation algorithm designed to enhance logit-based distillation. We introduce the concept of "perception", aiming to calibrate logits based on the model's representation capability. This concept addresses overconfidence issues in the logit-based distillation method while also introducing a novel method to distill knowledge from the teacher. It reconstructs the logits of a sample/instances by considering relationships with other samples in the batch. LumiNet excels on benchmarks like CIFAR-100, ImageNet, and MSCOCO, outperforming the leading feature-based methods, e.g., compared to KD with ResNet18 and MobileNetV2 on ImageNet, it shows improvements of 1.5% and 2.05%, respectively. Codes are available at https://github.com/ismail31416/LumiNet. |
| title | LumiNet: Perception-Driven Knowledge Distillation via Statistical Logit Calibration |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2310.03669 |