Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2503.09117 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913876304461824 |
|---|---|
| author | Wang, Yue Wang, Qizhou Liu, Feng Huang, Wei Du, Yali Du, Xiaojiang Han, Bo |
| author_facet | Wang, Yue Wang, Qizhou Liu, Feng Huang, Wei Du, Yali Du, Xiaojiang Han, Bo |
| contents | Large language model (LLM) unlearning has demonstrated its essential role in removing privacy and copyright-related responses, crucial for their legal and safe applications. However, the pursuit of complete unlearning often comes with substantial costs due to its compromises in their general functionality, leading to a notorious trade-off between unlearning and retention. It motivates this paper to explore enhanced unlearning schemes that can mitigate this trade-off. Specifically, we propose Gradient Rectified Unlearning (GRU), an improved framework that regulates the directions of gradient updates during the unlearning procedure such that their side impacts on other, unrelated responses can be minimized. GRU is easy and general to implement, demonstrating practical effectiveness across a variety of well-established unlearning benchmarks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_09117 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | GRU: Mitigating the Trade-off between Unlearning and Retention for LLMs Wang, Yue Wang, Qizhou Liu, Feng Huang, Wei Du, Yali Du, Xiaojiang Han, Bo Machine Learning Computation and Language Large language model (LLM) unlearning has demonstrated its essential role in removing privacy and copyright-related responses, crucial for their legal and safe applications. However, the pursuit of complete unlearning often comes with substantial costs due to its compromises in their general functionality, leading to a notorious trade-off between unlearning and retention. It motivates this paper to explore enhanced unlearning schemes that can mitigate this trade-off. Specifically, we propose Gradient Rectified Unlearning (GRU), an improved framework that regulates the directions of gradient updates during the unlearning procedure such that their side impacts on other, unrelated responses can be minimized. GRU is easy and general to implement, demonstrating practical effectiveness across a variety of well-established unlearning benchmarks. |
| title | GRU: Mitigating the Trade-off between Unlearning and Retention for LLMs |
| topic | Machine Learning Computation and Language |
| url | https://arxiv.org/abs/2503.09117 |