UIPE: Enhancing LLM Unlearning by Removing Knowledge Related to Forgetting Targets
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866913722546520064 |
|---|---|
| author | Wang, Wenyu Zhang, Mengqi Ye, Xiaotian Ren, Zhaochun Chen, Zhumin Ren, Pengjie |
| author_facet | Wang, Wenyu Zhang, Mengqi Ye, Xiaotian Ren, Zhaochun Chen, Zhumin Ren, Pengjie |
| contents | Large Language Models (LLMs) inevitably acquire harmful information during training on massive datasets. LLM unlearning aims to eliminate the influence of such harmful information while maintaining the model's overall performance. Existing unlearning methods, represented by gradient ascent-based approaches, primarily focus on forgetting target data while overlooking the crucial impact of logically related knowledge on the effectiveness of unlearning. In this paper, through both theoretical and experimental analyses, we first demonstrate that a key reason for the suboptimal unlearning performance is that models can reconstruct the target content through reasoning with logically related knowledge. To address this issue, we propose Unlearning Improvement via Parameter Extrapolation (UIPE), a method that removes knowledge highly correlated with the forgetting targets. Experimental results show that UIPE significantly enhances the performance of various mainstream LLM unlearning methods on the TOFU benchmark. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_04693 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | UIPE: Enhancing LLM Unlearning by Removing Knowledge Related to Forgetting Targets Wang, Wenyu Zhang, Mengqi Ye, Xiaotian Ren, Zhaochun Chen, Zhumin Ren, Pengjie Computation and Language Large Language Models (LLMs) inevitably acquire harmful information during training on massive datasets. LLM unlearning aims to eliminate the influence of such harmful information while maintaining the model's overall performance. Existing unlearning methods, represented by gradient ascent-based approaches, primarily focus on forgetting target data while overlooking the crucial impact of logically related knowledge on the effectiveness of unlearning. In this paper, through both theoretical and experimental analyses, we first demonstrate that a key reason for the suboptimal unlearning performance is that models can reconstruct the target content through reasoning with logically related knowledge. To address this issue, we propose Unlearning Improvement via Parameter Extrapolation (UIPE), a method that removes knowledge highly correlated with the forgetting targets. Experimental results show that UIPE significantly enhances the performance of various mainstream LLM unlearning methods on the TOFU benchmark. |
| title | UIPE: Enhancing LLM Unlearning by Removing Knowledge Related to Forgetting Targets |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2503.04693 |