Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2509.22263 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915832623267840 |
|---|---|
| author | Yang, Nakyeong Kim, Dong-Kyum Kwon, Jea Kim, Minsung Jung, Kyomin Cha, Meeyoung |
| author_facet | Yang, Nakyeong Kim, Dong-Kyum Kwon, Jea Kim, Minsung Jung, Kyomin Cha, Meeyoung |
| contents | Large language models trained on web-scale data can memorize private or sensitive knowledge, raising significant privacy risks. Although some unlearning methods mitigate these risks, they remain vulnerable to "relearning" during subsequent training, allowing a substantial portion of forgotten knowledge to resurface. In this paper, we show that widely used unlearning methods cause shallow alignment: instead of faithfully erasing target knowledge, they generate spurious unlearning neurons that amplify negative influence to hide it. To overcome this limitation, we introduce Ssiuu, a new class of unlearning methods that employs attribution-guided regularization to prevent spurious negative influence and faithfully remove target knowledge. Experimental results confirm that our method reliably erases target knowledge and outperforms strong baselines across two practical retraining scenarios: (1) adversarial injection of private data, and (2) benign attack using an instruction-following benchmark. Our findings highlight the necessity of robust and faithful unlearning methods for safe deployment of language models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_22263 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Erase or Hide? Suppressing Spurious Unlearning Neurons for Robust Unlearning Yang, Nakyeong Kim, Dong-Kyum Kwon, Jea Kim, Minsung Jung, Kyomin Cha, Meeyoung Machine Learning Large language models trained on web-scale data can memorize private or sensitive knowledge, raising significant privacy risks. Although some unlearning methods mitigate these risks, they remain vulnerable to "relearning" during subsequent training, allowing a substantial portion of forgotten knowledge to resurface. In this paper, we show that widely used unlearning methods cause shallow alignment: instead of faithfully erasing target knowledge, they generate spurious unlearning neurons that amplify negative influence to hide it. To overcome this limitation, we introduce Ssiuu, a new class of unlearning methods that employs attribution-guided regularization to prevent spurious negative influence and faithfully remove target knowledge. Experimental results confirm that our method reliably erases target knowledge and outperforms strong baselines across two practical retraining scenarios: (1) adversarial injection of private data, and (2) benign attack using an instruction-following benchmark. Our findings highlight the necessity of robust and faithful unlearning methods for safe deployment of language models. |
| title | Erase or Hide? Suppressing Spurious Unlearning Neurons for Robust Unlearning |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2509.22263 |