Swapped Logit Distillation via Bi-level Teacher Alignment
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866908375947673600 |
|---|---|
| author | Limantoro, Stephen Ekaputra Lin, Jhe-Hao Wang, Chih-Yu Tsai, Yi-Lung Shuai, Hong-Han Huang, Ching-Chun Cheng, Wen-Huang |
| author_facet | Limantoro, Stephen Ekaputra Lin, Jhe-Hao Wang, Chih-Yu Tsai, Yi-Lung Shuai, Hong-Han Huang, Ching-Chun Cheng, Wen-Huang |
| contents | Knowledge distillation (KD) compresses the network capacity by transferring knowledge from a large (teacher) network to a smaller one (student). It has been mainstream that the teacher directly transfers knowledge to the student with its original distribution, which can possibly lead to incorrect predictions. In this article, we propose a logit-based distillation via swapped logit processing, namely Swapped Logit Distillation (SLD). SLD is proposed under two assumptions: (1) the wrong prediction occurs when the prediction label confidence is not the maximum; (2) the "natural" limit of probability remains uncertain as the best value addition to the target cannot be determined. To address these issues, we propose a swapped logit processing scheme. Through this approach, we find that the swap method can be effectively extended to teacher and student outputs, transforming into two teachers. We further introduce loss scheduling to boost the performance of two teachers' alignment. Extensive experiments on image classification tasks demonstrate that SLD consistently performs best among previous state-of-the-art methods. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_20108 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Swapped Logit Distillation via Bi-level Teacher Alignment Limantoro, Stephen Ekaputra Lin, Jhe-Hao Wang, Chih-Yu Tsai, Yi-Lung Shuai, Hong-Han Huang, Ching-Chun Cheng, Wen-Huang Machine Learning Image and Video Processing Signal Processing Knowledge distillation (KD) compresses the network capacity by transferring knowledge from a large (teacher) network to a smaller one (student). It has been mainstream that the teacher directly transfers knowledge to the student with its original distribution, which can possibly lead to incorrect predictions. In this article, we propose a logit-based distillation via swapped logit processing, namely Swapped Logit Distillation (SLD). SLD is proposed under two assumptions: (1) the wrong prediction occurs when the prediction label confidence is not the maximum; (2) the "natural" limit of probability remains uncertain as the best value addition to the target cannot be determined. To address these issues, we propose a swapped logit processing scheme. Through this approach, we find that the swap method can be effectively extended to teacher and student outputs, transforming into two teachers. We further introduce loss scheduling to boost the performance of two teachers' alignment. Extensive experiments on image classification tasks demonstrate that SLD consistently performs best among previous state-of-the-art methods. |
| title | Swapped Logit Distillation via Bi-level Teacher Alignment |
| topic | Machine Learning Image and Video Processing Signal Processing |
| url | https://arxiv.org/abs/2504.20108 |