AI-KD: Adversarial learning and Implicit regularization for self-Knowledge Distillation
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2022
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866910376415723520 |
|---|---|
| author | Kim, Hyungmin Suh, Sungho Baek, Sunghyun Kim, Daehwan Jeong, Daun Cho, Hansang Kim, Junmo |
| author_facet | Kim, Hyungmin Suh, Sungho Baek, Sunghyun Kim, Daehwan Jeong, Daun Cho, Hansang Kim, Junmo |
| contents | We present a novel adversarial penalized self-knowledge distillation method, named adversarial learning and implicit regularization for self-knowledge distillation (AI-KD), which regularizes the training procedure by adversarial learning and implicit distillations. Our model not only distills the deterministic and progressive knowledge which are from the pre-trained and previous epoch predictive probabilities but also transfers the knowledge of the deterministic predictive distributions using adversarial learning. The motivation is that the self-knowledge distillation methods regularize the predictive probabilities with soft targets, but the exact distributions may be hard to predict. Our method deploys a discriminator to distinguish the distributions between the pre-trained and student models while the student model is trained to fool the discriminator in the trained procedure. Thus, the student model not only can learn the pre-trained model's predictive probabilities but also align the distributions between the pre-trained and student models. We demonstrate the effectiveness of the proposed method with network architectures on multiple datasets and show the proposed method achieves better performance than state-of-the-art methods. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2211_10938 |
| institution | arXiv |
| publishDate | 2022 |
| record_format | arxiv |
| spellingShingle | AI-KD: Adversarial learning and Implicit regularization for self-Knowledge Distillation Kim, Hyungmin Suh, Sungho Baek, Sunghyun Kim, Daehwan Jeong, Daun Cho, Hansang Kim, Junmo Computer Vision and Pattern Recognition Machine Learning We present a novel adversarial penalized self-knowledge distillation method, named adversarial learning and implicit regularization for self-knowledge distillation (AI-KD), which regularizes the training procedure by adversarial learning and implicit distillations. Our model not only distills the deterministic and progressive knowledge which are from the pre-trained and previous epoch predictive probabilities but also transfers the knowledge of the deterministic predictive distributions using adversarial learning. The motivation is that the self-knowledge distillation methods regularize the predictive probabilities with soft targets, but the exact distributions may be hard to predict. Our method deploys a discriminator to distinguish the distributions between the pre-trained and student models while the student model is trained to fool the discriminator in the trained procedure. Thus, the student model not only can learn the pre-trained model's predictive probabilities but also align the distributions between the pre-trained and student models. We demonstrate the effectiveness of the proposed method with network architectures on multiple datasets and show the proposed method achieves better performance than state-of-the-art methods. |
| title | AI-KD: Adversarial learning and Implicit regularization for self-Knowledge Distillation |
| topic | Computer Vision and Pattern Recognition Machine Learning |
| url | https://arxiv.org/abs/2211.10938 |