Knowledge Distillation via the Target-aware Transformer
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2022
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866929304224399360 |
|---|---|
| author | Lin, Sihao Xie, Hongwei Wang, Bing Yu, Kaicheng Chang, Xiaojun Liang, Xiaodan Wang, Gang |
| author_facet | Lin, Sihao Xie, Hongwei Wang, Bing Yu, Kaicheng Chang, Xiaojun Liang, Xiaodan Wang, Gang |
| contents | Knowledge distillation becomes a de facto standard to improve the performance of small neural networks. Most of the previous works propose to regress the representational features from the teacher to the student in a one-to-one spatial matching fashion. However, people tend to overlook the fact that, due to the architecture differences, the semantic information on the same spatial location usually vary. This greatly undermines the underlying assumption of the one-to-one distillation approach. To this end, we propose a novel one-to-all spatial matching knowledge distillation approach. Specifically, we allow each pixel of the teacher feature to be distilled to all spatial locations of the student features given its similarity, which is generated from a target-aware transformer. Our approach surpasses the state-of-the-art methods by a significant margin on various computer vision benchmarks, such as ImageNet, Pascal VOC and COCOStuff10k. Code is available at https://github.com/sihaoevery/TaT. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2205_10793 |
| institution | arXiv |
| publishDate | 2022 |
| record_format | arxiv |
| spellingShingle | Knowledge Distillation via the Target-aware Transformer Lin, Sihao Xie, Hongwei Wang, Bing Yu, Kaicheng Chang, Xiaojun Liang, Xiaodan Wang, Gang Computer Vision and Pattern Recognition Knowledge distillation becomes a de facto standard to improve the performance of small neural networks. Most of the previous works propose to regress the representational features from the teacher to the student in a one-to-one spatial matching fashion. However, people tend to overlook the fact that, due to the architecture differences, the semantic information on the same spatial location usually vary. This greatly undermines the underlying assumption of the one-to-one distillation approach. To this end, we propose a novel one-to-all spatial matching knowledge distillation approach. Specifically, we allow each pixel of the teacher feature to be distilled to all spatial locations of the student features given its similarity, which is generated from a target-aware transformer. Our approach surpasses the state-of-the-art methods by a significant margin on various computer vision benchmarks, such as ImageNet, Pascal VOC and COCOStuff10k. Code is available at https://github.com/sihaoevery/TaT. |
| title | Knowledge Distillation via the Target-aware Transformer |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2205.10793 |