Quantization-Robust LLM Unlearning via Low-Rank Adaptation
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2026
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866908940476874752 |
|---|---|
| author | Abitante, João Vitor Boer Pasquali, Joana Meneguzzo Garcia, Luan Fonseca de Oliveira, Ewerton Paula, Thomas da Silva Barros, Rodrigo C. Kupssinskü, Lucas S. |
| author_facet | Abitante, João Vitor Boer Pasquali, Joana Meneguzzo Garcia, Luan Fonseca de Oliveira, Ewerton Paula, Thomas da Silva Barros, Rodrigo C. Kupssinskü, Lucas S. |
| contents | Large Language Model (LLM) unlearning aims to remove targeted knowledge from a trained model, but practical deployments often require post-training quantization (PTQ) for efficient inference. However, aggressive low-bit PTQ can mask unlearning updates, causing quantized models to revert to pre-unlearning behavior. We show that standard full-parameter fine-tuning often induces parameter changes that are too small to survive 4-bit quantization. We propose quantization-robust unlearning via low-rank adaptation (LoRA): we freeze the base model and concentrate unlearning into trainable adapters so that the effective update is preserved after quantization. On Llama-2-7B evaluated with MUSE dataset (BOOKS and NEWS), LoRA improves 4-bit utility by up to 7.93 points (NPO+GDR on BOOKS: 50.17 to 58.10) and yields higher 4-bit utility on NEWS for GA+GDR (40.06 to 44.82, increase of 4.76). LoRA also substantially reduces privacy leakage under 4-bit PTQ, e.g., for GA+KLR on BOOKS, PrivLeak moves from -25.68 to -5.86 (closer to ideal 0), while maintaining strong forgetting (VerMem and KnowMem near 0). Thus, using LoRA for Machine Unlearning is beneficial for scenarios where quantization is necessary for model deployment. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_13151 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Quantization-Robust LLM Unlearning via Low-Rank Adaptation Abitante, João Vitor Boer Pasquali, Joana Meneguzzo Garcia, Luan Fonseca de Oliveira, Ewerton Paula, Thomas da Silva Barros, Rodrigo C. Kupssinskü, Lucas S. Machine Learning Computation and Language I.2.6; I.2.7 Large Language Model (LLM) unlearning aims to remove targeted knowledge from a trained model, but practical deployments often require post-training quantization (PTQ) for efficient inference. However, aggressive low-bit PTQ can mask unlearning updates, causing quantized models to revert to pre-unlearning behavior. We show that standard full-parameter fine-tuning often induces parameter changes that are too small to survive 4-bit quantization. We propose quantization-robust unlearning via low-rank adaptation (LoRA): we freeze the base model and concentrate unlearning into trainable adapters so that the effective update is preserved after quantization. On Llama-2-7B evaluated with MUSE dataset (BOOKS and NEWS), LoRA improves 4-bit utility by up to 7.93 points (NPO+GDR on BOOKS: 50.17 to 58.10) and yields higher 4-bit utility on NEWS for GA+GDR (40.06 to 44.82, increase of 4.76). LoRA also substantially reduces privacy leakage under 4-bit PTQ, e.g., for GA+KLR on BOOKS, PrivLeak moves from -25.68 to -5.86 (closer to ideal 0), while maintaining strong forgetting (VerMem and KnowMem near 0). Thus, using LoRA for Machine Unlearning is beneficial for scenarios where quantization is necessary for model deployment. |
| title | Quantization-Robust LLM Unlearning via Low-Rank Adaptation |
| topic | Machine Learning Computation and Language I.2.6; I.2.7 |
| url | https://arxiv.org/abs/2602.13151 |