Saved in:
| Main Authors: | , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2501.12619 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912233000271872 |
|---|---|
| author | Lee, Sunbowen Zhou, Junting Ao, Chang Li, Kaige Du, Xinrun He, Sirui Wu, Haihong Liu, Tianci Liu, Jiaheng Alinejad-Rokny, Hamid Yang, Min Liang, Yitao Wen, Zhoufutu Ni, Shiwen |
| author_facet | Lee, Sunbowen Zhou, Junting Ao, Chang Li, Kaige Du, Xinrun He, Sirui Wu, Haihong Liu, Tianci Liu, Jiaheng Alinejad-Rokny, Hamid Yang, Min Liang, Yitao Wen, Zhoufutu Ni, Shiwen |
| contents | Model distillation is a fundamental technique in building large language models (LLMs), transferring knowledge from a teacher model to a student model. However, distillation can lead to model homogenization, reducing diversity among models and impairing their ability to robustly handle complex or novel tasks. These limitations underscore the need to systematically quantify the distillation process and its impact. In this work, we propose a framework to evaluate and quantify model distillation. Our method addresses two key aspects: (1) Identifying identity cognition contradictions to assess discrepancies in how models perceive and represent identity-related information, and (2) Analyzing multi-granularity response similarities across models to measure the extent of homogenization. Experimental results demonstrate two key insights: (1) Well-known closed-source and open-source LLMs usually exhibit high distillation degrees, except for Claude, Doubao, and Gemini. (2) Base LLMs show higher distillation degrees compared to aligned LLMs. By offering a systematic approach to improve the transparency of LLM data distillation, we call for LLMs with more independent development and more transparent technical reports to improve LLMs' robustness and safety. The code and data are available under https://github.com/Aegis1863/LLMs-Distillation-Quantification. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_12619 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Quantification of Large Language Model Distillation Lee, Sunbowen Zhou, Junting Ao, Chang Li, Kaige Du, Xinrun He, Sirui Wu, Haihong Liu, Tianci Liu, Jiaheng Alinejad-Rokny, Hamid Yang, Min Liang, Yitao Wen, Zhoufutu Ni, Shiwen Computation and Language Model distillation is a fundamental technique in building large language models (LLMs), transferring knowledge from a teacher model to a student model. However, distillation can lead to model homogenization, reducing diversity among models and impairing their ability to robustly handle complex or novel tasks. These limitations underscore the need to systematically quantify the distillation process and its impact. In this work, we propose a framework to evaluate and quantify model distillation. Our method addresses two key aspects: (1) Identifying identity cognition contradictions to assess discrepancies in how models perceive and represent identity-related information, and (2) Analyzing multi-granularity response similarities across models to measure the extent of homogenization. Experimental results demonstrate two key insights: (1) Well-known closed-source and open-source LLMs usually exhibit high distillation degrees, except for Claude, Doubao, and Gemini. (2) Base LLMs show higher distillation degrees compared to aligned LLMs. By offering a systematic approach to improve the transparency of LLM data distillation, we call for LLMs with more independent development and more transparent technical reports to improve LLMs' robustness and safety. The code and data are available under https://github.com/Aegis1863/LLMs-Distillation-Quantification. |
| title | Quantification of Large Language Model Distillation |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2501.12619 |