Knowledge Distillation and Dataset Distillation of Large Language Models: Emerging Trends, Challenges, and Future Directions
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908745408184320 |
|---|---|
| author | Fang, Luyang Yu, Xiaowei Cai, Jiazhang Chen, Yongkai Wu, Shushan Liu, Zhengliang Yang, Zhenyuan Lu, Haoran Gong, Xilin Liu, Yufang Ma, Terry Ruan, Wei Abbasi, Ali Zhang, Jing Wang, Tao Latif, Ehsan You, Weihang Jiang, Hanqi Liu, Wei Zhang, Wei Kolouri, Soheil Zhai, Xiaoming Zhu, Dajiang Zhong, Wenxuan Liu, Tianming Ma, Ping |
| author_facet | Fang, Luyang Yu, Xiaowei Cai, Jiazhang Chen, Yongkai Wu, Shushan Liu, Zhengliang Yang, Zhenyuan Lu, Haoran Gong, Xilin Liu, Yufang Ma, Terry Ruan, Wei Abbasi, Ali Zhang, Jing Wang, Tao Latif, Ehsan You, Weihang Jiang, Hanqi Liu, Wei Zhang, Wei Kolouri, Soheil Zhai, Xiaoming Zhu, Dajiang Zhong, Wenxuan Liu, Tianming Ma, Ping |
| contents | The exponential growth of Large Language Models (LLMs) continues to highlight the need for efficient strategies to meet ever-expanding computational and data demands. This survey provides a comprehensive analysis of two complementary paradigms: Knowledge Distillation (KD) and Dataset Distillation (DD), both aimed at compressing LLMs while preserving their advanced reasoning capabilities and linguistic diversity. We first examine key methodologies in KD, such as task-specific alignment, rationale-based training, and multi-teacher frameworks, alongside DD techniques that synthesize compact, high-impact datasets through optimization-based gradient matching, latent space regularization, and generative synthesis. Building on these foundations, we explore how integrating KD and DD can produce more effective and scalable compression strategies. Together, these approaches address persistent challenges in model scalability, architectural heterogeneity, and the preservation of emergent LLM abilities. We further highlight applications across domains such as healthcare and education, where distillation enables efficient deployment without sacrificing performance. Despite substantial progress, open challenges remain in preserving emergent reasoning and linguistic diversity, enabling efficient adaptation to continually evolving teacher models and datasets, and establishing comprehensive evaluation protocols. By synthesizing methodological innovations, theoretical foundations, and practical insights, our survey charts a path toward sustainable, resource-efficient LLMs through the tighter integration of KD and DD principles. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_14772 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Knowledge Distillation and Dataset Distillation of Large Language Models: Emerging Trends, Challenges, and Future Directions Fang, Luyang Yu, Xiaowei Cai, Jiazhang Chen, Yongkai Wu, Shushan Liu, Zhengliang Yang, Zhenyuan Lu, Haoran Gong, Xilin Liu, Yufang Ma, Terry Ruan, Wei Abbasi, Ali Zhang, Jing Wang, Tao Latif, Ehsan You, Weihang Jiang, Hanqi Liu, Wei Zhang, Wei Kolouri, Soheil Zhai, Xiaoming Zhu, Dajiang Zhong, Wenxuan Liu, Tianming Ma, Ping Computation and Language Machine Learning The exponential growth of Large Language Models (LLMs) continues to highlight the need for efficient strategies to meet ever-expanding computational and data demands. This survey provides a comprehensive analysis of two complementary paradigms: Knowledge Distillation (KD) and Dataset Distillation (DD), both aimed at compressing LLMs while preserving their advanced reasoning capabilities and linguistic diversity. We first examine key methodologies in KD, such as task-specific alignment, rationale-based training, and multi-teacher frameworks, alongside DD techniques that synthesize compact, high-impact datasets through optimization-based gradient matching, latent space regularization, and generative synthesis. Building on these foundations, we explore how integrating KD and DD can produce more effective and scalable compression strategies. Together, these approaches address persistent challenges in model scalability, architectural heterogeneity, and the preservation of emergent LLM abilities. We further highlight applications across domains such as healthcare and education, where distillation enables efficient deployment without sacrificing performance. Despite substantial progress, open challenges remain in preserving emergent reasoning and linguistic diversity, enabling efficient adaptation to continually evolving teacher models and datasets, and establishing comprehensive evaluation protocols. By synthesizing methodological innovations, theoretical foundations, and practical insights, our survey charts a path toward sustainable, resource-efficient LLMs through the tighter integration of KD and DD principles. |
| title | Knowledge Distillation and Dataset Distillation of Large Language Models: Emerging Trends, Challenges, and Future Directions |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2504.14772 |