Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2407.18041 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917733216550912 |
|---|---|
| author | Hamidi, Shayan Mohajer Deng, Xizhen Tan, Renhao Ye, Linfeng Salamah, Ahmed Hussein |
| author_facet | Hamidi, Shayan Mohajer Deng, Xizhen Tan, Renhao Ye, Linfeng Salamah, Ahmed Hussein |
| contents | Recently, it was shown that the role of the teacher in knowledge distillation (KD) is to provide the student with an estimate of the true Bayes conditional probability density (BCPD). Notably, the new findings propose that the student's error rate can be upper-bounded by the mean squared error (MSE) between the teacher's output and BCPD. Consequently, to enhance KD efficacy, the teacher should be trained such that its output is close to BCPD in MSE sense. This paper elucidates that training the teacher model with MSE loss equates to minimizing the MSE between its output and BCPD, aligning with its core responsibility of providing the student with a BCPD estimate closely resembling it in MSE terms. In this respect, through a comprehensive set of experiments, we demonstrate that substituting the conventional teacher trained with cross-entropy loss with one trained using MSE loss in state-of-the-art KD methods consistently boosts the student's accuracy, resulting in improvements of up to 2.6\%. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_18041 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | How to Train the Teacher Model for Effective Knowledge Distillation Hamidi, Shayan Mohajer Deng, Xizhen Tan, Renhao Ye, Linfeng Salamah, Ahmed Hussein Machine Learning Recently, it was shown that the role of the teacher in knowledge distillation (KD) is to provide the student with an estimate of the true Bayes conditional probability density (BCPD). Notably, the new findings propose that the student's error rate can be upper-bounded by the mean squared error (MSE) between the teacher's output and BCPD. Consequently, to enhance KD efficacy, the teacher should be trained such that its output is close to BCPD in MSE sense. This paper elucidates that training the teacher model with MSE loss equates to minimizing the MSE between its output and BCPD, aligning with its core responsibility of providing the student with a BCPD estimate closely resembling it in MSE terms. In this respect, through a comprehensive set of experiments, we demonstrate that substituting the conventional teacher trained with cross-entropy loss with one trained using MSE loss in state-of-the-art KD methods consistently boosts the student's accuracy, resulting in improvements of up to 2.6\%. |
| title | How to Train the Teacher Model for Effective Knowledge Distillation |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2407.18041 |