Saved in:
| Main Author: | |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2506.16170 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911106574843904 |
|---|---|
| author | Singh, Simardeep |
| author_facet | Singh, Simardeep |
| contents | Large language models (LLMs) are known to memorize parts of their training data, raising important concerns around privacy and security. While previous research has focused on studying memorization in pre-trained models, much less is known about how knowledge distillation (KD) affects memorization.In this study, we explore how different KD methods influence the memorization of fine-tuned task data when a large teacher model is distilled into smaller student variants.This study demonstrates that distilling a larger teacher model, fine-tuned on a dataset, into a smaller variant not only lowers computational costs and model size but also significantly reduces the memorization risks compared to standard fine-tuning approaches. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_16170 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | From Teacher to Student: Tracking Memorization Through Model Distillation Singh, Simardeep Machine Learning Artificial Intelligence Large language models (LLMs) are known to memorize parts of their training data, raising important concerns around privacy and security. While previous research has focused on studying memorization in pre-trained models, much less is known about how knowledge distillation (KD) affects memorization.In this study, we explore how different KD methods influence the memorization of fine-tuned task data when a large teacher model is distilled into smaller student variants.This study demonstrates that distilling a larger teacher model, fine-tuned on a dataset, into a smaller variant not only lowers computational costs and model size but also significantly reduces the memorization risks compared to standard fine-tuning approaches. |
| title | From Teacher to Student: Tracking Memorization Through Model Distillation |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2506.16170 |