Data Compressibility Quantifies LLM Memorization
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914487302356992 |
|---|---|
| author | Huang, Yizhan Yang, Zhe Chen, Meifang Nianchen, Huang Zhang, Jianping Lyu, Michael R. |
| author_facet | Huang, Yizhan Yang, Zhe Chen, Meifang Nianchen, Huang Zhang, Jianping Lyu, Michael R. |
| contents | Large Language Models (LLMs) are known to memorize portions of their training data, sometimes even reproduce content verbatim when prompted appropriately. Despite substantial interest, existing LLM memorization research has offered limited insight into how training data influences memorization and largely lacks quantitative characterization. In this work, we build upon the line of research that seeks to quantify memorization through data compressibility. We analyze why prior attempts fail to yield a reliable quantitative measure and show that a surprisingly simple shift from instance-level to set-level metrics uncovers a robust phenomenon, which we term the \textit{Entropy--Memorization (EM) Linearity}. This law states that a set-level data entropy estimator exhibits a linear correlation with memorization scores. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_06056 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Data Compressibility Quantifies LLM Memorization Huang, Yizhan Yang, Zhe Chen, Meifang Nianchen, Huang Zhang, Jianping Lyu, Michael R. Computation and Language Artificial Intelligence Large Language Models (LLMs) are known to memorize portions of their training data, sometimes even reproduce content verbatim when prompted appropriately. Despite substantial interest, existing LLM memorization research has offered limited insight into how training data influences memorization and largely lacks quantitative characterization. In this work, we build upon the line of research that seeks to quantify memorization through data compressibility. We analyze why prior attempts fail to yield a reliable quantitative measure and show that a surprisingly simple shift from instance-level to set-level metrics uncovers a robust phenomenon, which we term the \textit{Entropy--Memorization (EM) Linearity}. This law states that a set-level data entropy estimator exhibits a linear correlation with memorization scores. |
| title | Data Compressibility Quantifies LLM Memorization |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2507.06056 |