Multi-Teacher Knowledge Distillation with Reinforcement Learning for Visual Recognition
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866916630343188480 |
|---|---|
| author | Yang, Chuanguang Yu, Xinqiang Yang, Han An, Zhulin Yu, Chengqing Huang, Libo Xu, Yongjun |
| author_facet | Yang, Chuanguang Yu, Xinqiang Yang, Han An, Zhulin Yu, Chengqing Huang, Libo Xu, Yongjun |
| contents | Multi-teacher Knowledge Distillation (KD) transfers diverse knowledge from a teacher pool to a student network. The core problem of multi-teacher KD is how to balance distillation strengths among various teachers. Most existing methods often develop weighting strategies from an individual perspective of teacher performance or teacher-student gaps, lacking comprehensive information for guidance. This paper proposes Multi-Teacher Knowledge Distillation with Reinforcement Learning (MTKD-RL) to optimize multi-teacher weights. In this framework, we construct both teacher performance and teacher-student gaps as state information to an agent. The agent outputs the teacher weight and can be updated by the return reward from the student. MTKD-RL reinforces the interaction between the student and teacher using an agent in an RL-based decision mechanism, achieving better matching capability with more meaningful weights. Experimental results on visual recognition tasks, including image classification, object detection, and semantic segmentation tasks, demonstrate that MTKD-RL achieves state-of-the-art performance compared to the existing multi-teacher KD works. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_18510 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Multi-Teacher Knowledge Distillation with Reinforcement Learning for Visual Recognition Yang, Chuanguang Yu, Xinqiang Yang, Han An, Zhulin Yu, Chengqing Huang, Libo Xu, Yongjun Computer Vision and Pattern Recognition Multi-teacher Knowledge Distillation (KD) transfers diverse knowledge from a teacher pool to a student network. The core problem of multi-teacher KD is how to balance distillation strengths among various teachers. Most existing methods often develop weighting strategies from an individual perspective of teacher performance or teacher-student gaps, lacking comprehensive information for guidance. This paper proposes Multi-Teacher Knowledge Distillation with Reinforcement Learning (MTKD-RL) to optimize multi-teacher weights. In this framework, we construct both teacher performance and teacher-student gaps as state information to an agent. The agent outputs the teacher weight and can be updated by the return reward from the student. MTKD-RL reinforces the interaction between the student and teacher using an agent in an RL-based decision mechanism, achieving better matching capability with more meaningful weights. Experimental results on visual recognition tasks, including image classification, object detection, and semantic segmentation tasks, demonstrate that MTKD-RL achieves state-of-the-art performance compared to the existing multi-teacher KD works. |
| title | Multi-Teacher Knowledge Distillation with Reinforcement Learning for Visual Recognition |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2502.18510 |