AdaKD: Dynamic Knowledge Distillation of ASR models using Adaptive Loss Weighting
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866917664420528128 |
|---|---|
| author | Ganguly, Shreyan Nayak, Roshan Rao, Rakshith Deb, Ujan AP, Prathosh |
| author_facet | Ganguly, Shreyan Nayak, Roshan Rao, Rakshith Deb, Ujan AP, Prathosh |
| contents | Knowledge distillation, a widely used model compression technique, works on the basis of transferring knowledge from a cumbersome teacher model to a lightweight student model. The technique involves jointly optimizing the task specific and knowledge distillation losses with a weight assigned to them. Despite these weights playing a crucial role in the performance of the distillation process, current methods provide equal weight to both losses, leading to suboptimal performance. In this paper, we propose Adaptive Knowledge Distillation, a novel technique inspired by curriculum learning to adaptively weigh the losses at instance level. This technique goes by the notion that sample difficulty increases with teacher loss. Our method follows a plug-and-play paradigm that can be applied on top of any task-specific and distillation objectives. Experiments show that our method performs better than conventional knowledge distillation method and existing instance-level loss functions. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_08019 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | AdaKD: Dynamic Knowledge Distillation of ASR models using Adaptive Loss Weighting Ganguly, Shreyan Nayak, Roshan Rao, Rakshith Deb, Ujan AP, Prathosh Machine Learning Artificial Intelligence Knowledge distillation, a widely used model compression technique, works on the basis of transferring knowledge from a cumbersome teacher model to a lightweight student model. The technique involves jointly optimizing the task specific and knowledge distillation losses with a weight assigned to them. Despite these weights playing a crucial role in the performance of the distillation process, current methods provide equal weight to both losses, leading to suboptimal performance. In this paper, we propose Adaptive Knowledge Distillation, a novel technique inspired by curriculum learning to adaptively weigh the losses at instance level. This technique goes by the notion that sample difficulty increases with teacher loss. Our method follows a plug-and-play paradigm that can be applied on top of any task-specific and distillation objectives. Experiments show that our method performs better than conventional knowledge distillation method and existing instance-level loss functions. |
| title | AdaKD: Dynamic Knowledge Distillation of ASR models using Adaptive Loss Weighting |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2405.08019 |