Cover Image

Memorization Dynamics in Knowledge Distillation for Language Models

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Borkar, Jaydeep, Chadha, Karan, Mireshghallah, Niloofar, Zhang, Yuchen, Veliche, Irina-Elena, Mitra, Archi, Smith, David A., Xu, Zheng, Garcia-Olano, Diego
Format:	Preprint
Published:	2026
Subjects:	Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Privacy Ripple Effects from Adding or Removing Personal Information in Language Model Training
by: Borkar, Jaydeep, et al.
Published: (2025)

Position: Privacy Is Not Just Memorization!
by: Mireshghallah, Niloofar, et al.
Published: (2025)

Mind the Gap: Analyzing Lacunae with Transformer-Based Transcription
by: Borkar, Jaydeep, et al.
Published: (2024)

Reinforcement Learning Improves Traversal of Hierarchical Knowledge in LLMs
by: Zhang, Renfei, et al.
Published: (2025)

Synthetic Data Can Mislead Evaluations: Membership Inference as Machine Text Detection
by: Naseh, Ali, et al.
Published: (2025)

Differentially Private Learning Needs Better Model Initialization and Self-Distillation
by: Ngong, Ivoline C., et al.
Published: (2024)

Bob's Confetti: Phonetic Memorization Attacks in Music and Video Generation
by: Roh, Jaechul, et al.
Published: (2025)

Alpaca against Vicuna: Using LLMs to Uncover Memorization of LLMs
by: Kassem, Aly M., et al.
Published: (2024)

Alignment Whack-a-Mole : Finetuning Activates Verbatim Recall of Copyrighted Books in Large Language Models
by: Liu, Xinyue, et al.
Published: (2026)

Cross-Lingual Knowledge Distillation for Answer Sentence Selection in Low-Resource Languages
by: Gupta, Shivanshu, et al.
Published: (2023)

Smaller Language Models are Better Black-box Machine-Generated Text Detectors
by: Mireshghallah, Niloofar, et al.
Published: (2023)

Information-Guided Identification of Training Data Imprint in (Proprietary) Large Language Models
by: Ravichander, Abhilasha, et al.
Published: (2025)

Trust No Bot: Discovering Personal Disclosures in Human-LLM Conversations in the Wild
by: Mireshghallah, Niloofar, et al.
Published: (2024)

Recite, Reconstruct, Recollect: Memorization in LMs as a Multifaceted Phenomenon
by: Prashanth, USVSN Sai, et al.
Published: (2024)

Memorization Inheritance in Sequence-Level Knowledge Distillation for Neural Machine Translation
by: Dankers, Verna, et al.
Published: (2025)

Developing Story: Case Studies of Generative AI's Use in Journalism
by: Brigham, Natalie Grace, et al.
Published: (2024)

Memorization and Knowledge Injection in Gated LLMs
by: Pan, Xu, et al.
Published: (2025)

Can LLMs Keep a Secret? Testing Privacy Implications of Language Models via Contextual Integrity Theory
by: Mireshghallah, Niloofar, et al.
Published: (2023)

Boundary-targeted Membership Inference Attacks on Safety Classifiers
by: Hughes, Anthony, et al.
Published: (2026)

Do Membership Inference Attacks Work on Large Language Models?
by: Duan, Michael, et al.
Published: (2024)

On Memorization of Large Language Models in Logical Reasoning
by: Xie, Chulin, et al.
Published: (2024)

Knowledge Distillation for Large Language Models
by: La Torre, Alejandro Paredes, et al.
Published: (2026)

Retentive or Forgetful? Diving into the Knowledge Memorizing Mechanism of Language Models
by: Cao, Boxi, et al.
Published: (2023)

ParaPO: Aligning Language Models to Reduce Verbatim Reproduction of Pre-training Data
by: Chen, Tong, et al.
Published: (2025)

A Survey on Knowledge Distillation of Large Language Models
by: Xu, Xiaohan, et al.
Published: (2024)

Unlocking Memorization in Large Language Models with Dynamic Soft Prompting
by: Wang, Zhepeng, et al.
Published: (2024)

Private Memorization Editing: Turning Memorization into a Defense to Strengthen Data Privacy in Large Language Models
by: Ruzzetti, Elena Sofia, et al.
Published: (2025)

CopyBench: Measuring Literal and Non-Literal Reproduction of Copyright-Protected Text in Language Model Generation
by: Chen, Tong, et al.
Published: (2024)

Memorizing is Not Enough: Deep Knowledge Injection Through Reasoning
by: Xu, Ruoxi, et al.
Published: (2025)

HumorGen: Cognitive Synergy for Humor Generation in Large Language Models via Persona-Based Distillation
by: Ajayi, Edward, et al.
Published: (2026)

What Matters in Memorizing and Recalling Facts? Multifaceted Benchmarks for Knowledge Probing in Language Models
by: Zhao, Xin, et al.
Published: (2024)

SpikeBERT: A Language Spikformer Learned from BERT with Knowledge Distillation
by: Lv, Changze, et al.
Published: (2023)

WildTeaming at Scale: From In-the-Wild Jailbreaks to (Adversarially) Safer Language Models
by: Jiang, Liwei, et al.
Published: (2024)

Enhancing Cross-Tokenizer Knowledge Distillation with Contextual Dynamical Mapping
by: Chen, Yijie, et al.
Published: (2025)

Confidence Preservation Property in Knowledge Distillation Abstractions
by: Vengertsev, Dmitry, et al.
Published: (2024)

AI as Humanity's Salieri: Quantifying Linguistic Creativity of Language Models via Systematic Attribution of Machine Text against Web Text
by: Lu, Ximing, et al.
Published: (2024)

Spectrum Tuning: Post-Training for Distributional Coverage and In-Context Steerability
by: Sorensen, Taylor, et al.
Published: (2025)

To Memorize or to Retrieve: Scaling Laws for RAG-Considerate Pretraining
by: Singh, Karan, et al.
Published: (2026)

Continual Memorization of Factoids in Language Models
by: Chen, Howard, et al.
Published: (2024)

Dual-Space Knowledge Distillation for Large Language Models
by: Zhang, Songming, et al.
Published: (2024)