Exploring Knowledge Purification in Multi-Teacher Knowledge Distillation for LLMs
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2026
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866913023607701504 |
|---|---|
| author | Jin, Ruihan Shao, Pengpeng Wen, Zhengqi Wu, Jinyang Feng, Mingkuan Yang, Shuo Zhang, Chu Yuan Tao, Jianhua |
| author_facet | Jin, Ruihan Shao, Pengpeng Wen, Zhengqi Wu, Jinyang Feng, Mingkuan Yang, Shuo Zhang, Chu Yuan Tao, Jianhua |
| contents | Knowledge distillation has emerged as a pivotal technique for transferring knowledge from stronger large language models (LLMs) to smaller, more efficient models. However, traditional distillation approaches face challenges related to knowledge conflicts and high resource demands, particularly when leveraging multiple teacher models. In this paper, we introduce the concept of \textbf{Knowledge Purification}, which consolidates the rationales from multiple teacher LLMs into a single rationale, thereby mitigating conflicts and enhancing efficiency. To investigate the effectiveness of knowledge purification, we further propose five purification methods from various perspectives. Our experiments demonstrate that these methods not only improve the performance of the distilled model but also effectively alleviate knowledge conflicts. Moreover, router-based methods exhibit robust generalization capabilities, underscoring the potential of innovative purification techniques in optimizing multi-teacher distillation and facilitating the practical deployment of powerful yet lightweight models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_01064 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Exploring Knowledge Purification in Multi-Teacher Knowledge Distillation for LLMs Jin, Ruihan Shao, Pengpeng Wen, Zhengqi Wu, Jinyang Feng, Mingkuan Yang, Shuo Zhang, Chu Yuan Tao, Jianhua Computation and Language Knowledge distillation has emerged as a pivotal technique for transferring knowledge from stronger large language models (LLMs) to smaller, more efficient models. However, traditional distillation approaches face challenges related to knowledge conflicts and high resource demands, particularly when leveraging multiple teacher models. In this paper, we introduce the concept of \textbf{Knowledge Purification}, which consolidates the rationales from multiple teacher LLMs into a single rationale, thereby mitigating conflicts and enhancing efficiency. To investigate the effectiveness of knowledge purification, we further propose five purification methods from various perspectives. Our experiments demonstrate that these methods not only improve the performance of the distilled model but also effectively alleviate knowledge conflicts. Moreover, router-based methods exhibit robust generalization capabilities, underscoring the potential of innovative purification techniques in optimizing multi-teacher distillation and facilitating the practical deployment of powerful yet lightweight models. |
| title | Exploring Knowledge Purification in Multi-Teacher Knowledge Distillation for LLMs |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2602.01064 |