Saved in:
| Main Authors: | , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2506.20856 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915359574982656 |
|---|---|
| author | Wang, Fei Li, Baochun |
| author_facet | Wang, Fei Li, Baochun |
| contents | Memorization in large language models (LLMs) makes them vulnerable to data extraction attacks. While pre-training memorization has been extensively studied, fewer works have explored its impact in fine-tuning, particularly for LoRA fine-tuning, a widely adopted parameter-efficient method.
In this work, we re-examine memorization in fine-tuning and uncover a surprising divergence from prior findings across different fine-tuning strategies. Factors such as model scale and data duplication, which strongly influence memorization in pre-training and full fine-tuning, do not follow the same trend in LoRA fine-tuning. Using a more relaxed similarity-based memorization metric, we demonstrate that LoRA significantly reduces memorization risks compared to full fine-tuning, while still maintaining strong task performance. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_20856 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Leaner Training, Lower Leakage: Revisiting Memorization in LLM Fine-Tuning with LoRA Wang, Fei Li, Baochun Machine Learning Computation and Language Cryptography and Security Memorization in large language models (LLMs) makes them vulnerable to data extraction attacks. While pre-training memorization has been extensively studied, fewer works have explored its impact in fine-tuning, particularly for LoRA fine-tuning, a widely adopted parameter-efficient method. In this work, we re-examine memorization in fine-tuning and uncover a surprising divergence from prior findings across different fine-tuning strategies. Factors such as model scale and data duplication, which strongly influence memorization in pre-training and full fine-tuning, do not follow the same trend in LoRA fine-tuning. Using a more relaxed similarity-based memorization metric, we demonstrate that LoRA significantly reduces memorization risks compared to full fine-tuning, while still maintaining strong task performance. |
| title | Leaner Training, Lower Leakage: Revisiting Memorization in LLM Fine-Tuning with LoRA |
| topic | Machine Learning Computation and Language Cryptography and Security |
| url | https://arxiv.org/abs/2506.20856 |