Survey on Knowledge Distillation for Large Language Models: Methods, Evaluation, and Application
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911940264067072 |
|---|---|
| author | Yang, Chuanpeng Lu, Wang Zhu, Yao Wang, Yidong Chen, Qian Gao, Chenlong Yan, Bingjie Chen, Yiqiang |
| author_facet | Yang, Chuanpeng Lu, Wang Zhu, Yao Wang, Yidong Chen, Qian Gao, Chenlong Yan, Bingjie Chen, Yiqiang |
| contents | Large Language Models (LLMs) have showcased exceptional capabilities in various domains, attracting significant interest from both academia and industry. Despite their impressive performance, the substantial size and computational demands of LLMs pose considerable challenges for practical deployment, particularly in environments with limited resources. The endeavor to compress language models while maintaining their accuracy has become a focal point of research. Among the various methods, knowledge distillation has emerged as an effective technique to enhance inference speed without greatly compromising performance. This paper presents a thorough survey from three aspects: method, evaluation, and application, exploring knowledge distillation techniques tailored specifically for LLMs. Specifically, we divide the methods into white-box KD and black-box KD to better illustrate their differences. Furthermore, we also explored the evaluation tasks and distillation effects between different distillation methods, and proposed directions for future research. Through in-depth understanding of the latest advancements and practical applications, this survey provides valuable resources for researchers, paving the way for sustained progress in this field. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_01885 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Survey on Knowledge Distillation for Large Language Models: Methods, Evaluation, and Application Yang, Chuanpeng Lu, Wang Zhu, Yao Wang, Yidong Chen, Qian Gao, Chenlong Yan, Bingjie Chen, Yiqiang Computation and Language Artificial Intelligence Large Language Models (LLMs) have showcased exceptional capabilities in various domains, attracting significant interest from both academia and industry. Despite their impressive performance, the substantial size and computational demands of LLMs pose considerable challenges for practical deployment, particularly in environments with limited resources. The endeavor to compress language models while maintaining their accuracy has become a focal point of research. Among the various methods, knowledge distillation has emerged as an effective technique to enhance inference speed without greatly compromising performance. This paper presents a thorough survey from three aspects: method, evaluation, and application, exploring knowledge distillation techniques tailored specifically for LLMs. Specifically, we divide the methods into white-box KD and black-box KD to better illustrate their differences. Furthermore, we also explored the evaluation tasks and distillation effects between different distillation methods, and proposed directions for future research. Through in-depth understanding of the latest advancements and practical applications, this survey provides valuable resources for researchers, paving the way for sustained progress in this field. |
| title | Survey on Knowledge Distillation for Large Language Models: Methods, Evaluation, and Application |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2407.01885 |