Privacy Ripple Effects from Adding or Removing Personal Information in Language Model Training
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866909660124020736 |
|---|---|
| author | Borkar, Jaydeep Jagielski, Matthew Lee, Katherine Mireshghallah, Niloofar Smith, David A. Choquette-Choo, Christopher A. |
| author_facet | Borkar, Jaydeep Jagielski, Matthew Lee, Katherine Mireshghallah, Niloofar Smith, David A. Choquette-Choo, Christopher A. |
| contents | Due to the sensitive nature of personally identifiable information (PII), its owners may have the authority to control its inclusion or request its removal from large-language model (LLM) training. Beyond this, PII may be added or removed from training datasets due to evolving dataset curation techniques, because they were newly scraped for retraining, or because they were included in a new downstream fine-tuning stage. We find that the amount and ease of PII memorization is a dynamic property of a model that evolves throughout training pipelines and depends on commonly altered design choices. We characterize three such novel phenomena: (1) similar-appearing PII seen later in training can elicit memorization of earlier-seen sequences in what we call assisted memorization, and this is a significant factor (in our settings, up to 1/3); (2) adding PII can increase memorization of other PII significantly (in our settings, as much as $\approx\!7.5\times$); and (3) removing PII can lead to other PII being memorized. Model creators should consider these first- and second-order privacy risks when training models to avoid the risk of new PII regurgitation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_15680 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Privacy Ripple Effects from Adding or Removing Personal Information in Language Model Training Borkar, Jaydeep Jagielski, Matthew Lee, Katherine Mireshghallah, Niloofar Smith, David A. Choquette-Choo, Christopher A. Computation and Language Cryptography and Security Due to the sensitive nature of personally identifiable information (PII), its owners may have the authority to control its inclusion or request its removal from large-language model (LLM) training. Beyond this, PII may be added or removed from training datasets due to evolving dataset curation techniques, because they were newly scraped for retraining, or because they were included in a new downstream fine-tuning stage. We find that the amount and ease of PII memorization is a dynamic property of a model that evolves throughout training pipelines and depends on commonly altered design choices. We characterize three such novel phenomena: (1) similar-appearing PII seen later in training can elicit memorization of earlier-seen sequences in what we call assisted memorization, and this is a significant factor (in our settings, up to 1/3); (2) adding PII can increase memorization of other PII significantly (in our settings, as much as $\approx\!7.5\times$); and (3) removing PII can lead to other PII being memorized. Model creators should consider these first- and second-order privacy risks when training models to avoid the risk of new PII regurgitation. |
| title | Privacy Ripple Effects from Adding or Removing Personal Information in Language Model Training |
| topic | Computation and Language Cryptography and Security |
| url | https://arxiv.org/abs/2502.15680 |