The Privileged Students: On the Value of Initialization in Multilingual Knowledge Distillation
Fuente:
arXiv
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913574035652608 |
|---|---|
| author | Wibowo, Haryo Akbarianto Solorio, Thamar Aji, Alham Fikri |
| author_facet | Wibowo, Haryo Akbarianto Solorio, Thamar Aji, Alham Fikri |
| contents | Knowledge distillation (KD) has proven to be a successful strategy to improve the performance of smaller models in many NLP tasks. However, most of the work in KD only explores monolingual scenarios. In this paper, we investigate the value of KD in multilingual settings. We find the significance of KD and model initialization by analyzing how well the student model acquires multilingual knowledge from the teacher model. Our proposed method emphasizes copying the teacher model's weights directly to the student model to enhance initialization. Our findings show that model initialization using copy-weight from the fine-tuned teacher contributes the most compared to the distillation process itself across various multilingual settings. Furthermore, we demonstrate that efficient weight initialization preserves multilingual capabilities even in low-resource scenarios. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_16524 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | The Privileged Students: On the Value of Initialization in Multilingual Knowledge Distillation Wibowo, Haryo Akbarianto Solorio, Thamar Aji, Alham Fikri Computation and Language 68T50 Knowledge distillation (KD) has proven to be a successful strategy to improve the performance of smaller models in many NLP tasks. However, most of the work in KD only explores monolingual scenarios. In this paper, we investigate the value of KD in multilingual settings. We find the significance of KD and model initialization by analyzing how well the student model acquires multilingual knowledge from the teacher model. Our proposed method emphasizes copying the teacher model's weights directly to the student model to enhance initialization. Our findings show that model initialization using copy-weight from the fine-tuned teacher contributes the most compared to the distillation process itself across various multilingual settings. Furthermore, we demonstrate that efficient weight initialization preserves multilingual capabilities even in low-resource scenarios. |
| title | The Privileged Students: On the Value of Initialization in Multilingual Knowledge Distillation |
| topic | Computation and Language 68T50 |
| url | https://arxiv.org/abs/2406.16524 |