An Empirical Investigation into the Effect of Parameter Choices in Knowledge Distillation
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866929247419891712 |
|---|---|
| author | Sultan, Md Arafat Trivedi, Aashka Awasthy, Parul Sil, Avirup |
| author_facet | Sultan, Md Arafat Trivedi, Aashka Awasthy, Parul Sil, Avirup |
| contents | We present a large-scale empirical study of how choices of configuration parameters affect performance in knowledge distillation (KD). An example of such a KD parameter is the measure of distance between the predictions of the teacher and the student, common choices for which include the mean squared error (MSE) and the KL-divergence. Although scattered efforts have been made to understand the differences between such options, the KD literature still lacks a systematic study on their general effect on student performance. We take an empirical approach to this question in this paper, seeking to find out the extent to which such choices influence student performance across 13 datasets from 4 NLP tasks and 3 student sizes. We quantify the cost of making sub-optimal choices and identify a single configuration that performs well across the board. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2401_06356 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | An Empirical Investigation into the Effect of Parameter Choices in Knowledge Distillation Sultan, Md Arafat Trivedi, Aashka Awasthy, Parul Sil, Avirup Machine Learning We present a large-scale empirical study of how choices of configuration parameters affect performance in knowledge distillation (KD). An example of such a KD parameter is the measure of distance between the predictions of the teacher and the student, common choices for which include the mean squared error (MSE) and the KL-divergence. Although scattered efforts have been made to understand the differences between such options, the KD literature still lacks a systematic study on their general effect on student performance. We take an empirical approach to this question in this paper, seeking to find out the extent to which such choices influence student performance across 13 datasets from 4 NLP tasks and 3 student sizes. We quantify the cost of making sub-optimal choices and identify a single configuration that performs well across the board. |
| title | An Empirical Investigation into the Effect of Parameter Choices in Knowledge Distillation |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2401.06356 |