Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2305.09651 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910448466526208 |
|---|---|
| author | Ren, Yuxin Zhong, Zihan Shi, Xingjian Zhu, Yi Yuan, Chun Li, Mu |
| author_facet | Ren, Yuxin Zhong, Zihan Shi, Xingjian Zhu, Yi Yuan, Chun Li, Mu |
| contents | It has been commonly observed that a teacher model with superior performance does not necessarily result in a stronger student, highlighting a discrepancy between current teacher training practices and effective knowledge transfer. In order to enhance the guidance of the teacher training process, we introduce the concept of distillation influence to determine the impact of distillation from each training sample on the student's generalization ability. In this paper, we propose Learning Good Teacher Matters (LGTM), an efficient training technique for incorporating distillation influence into the teacher's learning process. By prioritizing samples that are likely to enhance the student's generalization ability, our LGTM outperforms 10 common knowledge distillation baselines on 6 text classification tasks in the GLUE benchmark. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2305_09651 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Tailoring Instructions to Student's Learning Levels Boosts Knowledge Distillation Ren, Yuxin Zhong, Zihan Shi, Xingjian Zhu, Yi Yuan, Chun Li, Mu Computation and Language Machine Learning It has been commonly observed that a teacher model with superior performance does not necessarily result in a stronger student, highlighting a discrepancy between current teacher training practices and effective knowledge transfer. In order to enhance the guidance of the teacher training process, we introduce the concept of distillation influence to determine the impact of distillation from each training sample on the student's generalization ability. In this paper, we propose Learning Good Teacher Matters (LGTM), an efficient training technique for incorporating distillation influence into the teacher's learning process. By prioritizing samples that are likely to enhance the student's generalization ability, our LGTM outperforms 10 common knowledge distillation baselines on 6 text classification tasks in the GLUE benchmark. |
| title | Tailoring Instructions to Student's Learning Levels Boosts Knowledge Distillation |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2305.09651 |