DDK: Distilling Domain Knowledge for Efficient Large Language Models
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866916333066649600 |
|---|---|
| author | Liu, Jiaheng Zhang, Chenchen Guo, Jinyang Zhang, Yuanxing Que, Haoran Deng, Ken Bai, Zhiqi Liu, Jie Zhang, Ge Wang, Jiakai Wu, Yanan Liu, Congnan Su, Wenbo Wang, Jiamang Qu, Lin Zheng, Bo |
| author_facet | Liu, Jiaheng Zhang, Chenchen Guo, Jinyang Zhang, Yuanxing Que, Haoran Deng, Ken Bai, Zhiqi Liu, Jie Zhang, Ge Wang, Jiakai Wu, Yanan Liu, Congnan Su, Wenbo Wang, Jiamang Qu, Lin Zheng, Bo |
| contents | Despite the advanced intelligence abilities of large language models (LLMs) in various applications, they still face significant computational and storage demands. Knowledge Distillation (KD) has emerged as an effective strategy to improve the performance of a smaller LLM (i.e., the student model) by transferring knowledge from a high-performing LLM (i.e., the teacher model). Prevailing techniques in LLM distillation typically use a black-box model API to generate high-quality pretrained and aligned datasets, or utilize white-box distillation by altering the loss function to better transfer knowledge from the teacher LLM. However, these methods ignore the knowledge differences between the student and teacher LLMs across domains. This results in excessive focus on domains with minimal performance gaps and insufficient attention to domains with large gaps, reducing overall performance. In this paper, we introduce a new LLM distillation framework called DDK, which dynamically adjusts the composition of the distillation dataset in a smooth manner according to the domain performance differences between the teacher and student models, making the distillation process more stable and effective. Extensive evaluations show that DDK significantly improves the performance of student models, outperforming both continuously pretrained baselines and existing knowledge distillation methods by a large margin. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_16154 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | DDK: Distilling Domain Knowledge for Efficient Large Language Models Liu, Jiaheng Zhang, Chenchen Guo, Jinyang Zhang, Yuanxing Que, Haoran Deng, Ken Bai, Zhiqi Liu, Jie Zhang, Ge Wang, Jiakai Wu, Yanan Liu, Congnan Su, Wenbo Wang, Jiamang Qu, Lin Zheng, Bo Computation and Language Despite the advanced intelligence abilities of large language models (LLMs) in various applications, they still face significant computational and storage demands. Knowledge Distillation (KD) has emerged as an effective strategy to improve the performance of a smaller LLM (i.e., the student model) by transferring knowledge from a high-performing LLM (i.e., the teacher model). Prevailing techniques in LLM distillation typically use a black-box model API to generate high-quality pretrained and aligned datasets, or utilize white-box distillation by altering the loss function to better transfer knowledge from the teacher LLM. However, these methods ignore the knowledge differences between the student and teacher LLMs across domains. This results in excessive focus on domains with minimal performance gaps and insufficient attention to domains with large gaps, reducing overall performance. In this paper, we introduce a new LLM distillation framework called DDK, which dynamically adjusts the composition of the distillation dataset in a smooth manner according to the domain performance differences between the teacher and student models, making the distillation process more stable and effective. Extensive evaluations show that DDK significantly improves the performance of student models, outperforming both continuously pretrained baselines and existing knowledge distillation methods by a large margin. |
| title | DDK: Distilling Domain Knowledge for Efficient Large Language Models |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2407.16154 |