Go Beyond Your Means: Unlearning with Per-Sample Gradient Orthogonalization
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918379113152512 |
|---|---|
| author | Shamsian, Aviv Shaar, Eitan Navon, Aviv Chechik, Gal Fetaya, Ethan |
| author_facet | Shamsian, Aviv Shaar, Eitan Navon, Aviv Chechik, Gal Fetaya, Ethan |
| contents | Machine unlearning aims to remove the influence of problematic training data after a model has been trained. The primary challenge in machine unlearning is ensuring that the process effectively removes specified data without compromising the model's overall performance on the remaining dataset. Many existing machine unlearning methods address this challenge by carefully balancing gradient ascent on the `unlearn' data with the gradient descent on a `retain' set that represents the training data. However, in many cases the training dataset is not fully available when we wish to unlearn some concepts, because models are released without their training datasets, and one may only have access to a $\textit{small part of a training set}$. Here, we propose OrthoGrad, a novel approach that mitigates interference between the unlearn set and a small retain set rather than competing ascent and descent processes. Our method projects the gradient of the unlearn set onto the subspace orthogonal to all gradients in the retain batch, effectively avoiding any gradient interference. We demonstrate the effectiveness of OrthoGrad on multiple machine unlearning benchmarks, including automatic speech recognition, outperforming competing methods. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_02312 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Go Beyond Your Means: Unlearning with Per-Sample Gradient Orthogonalization Shamsian, Aviv Shaar, Eitan Navon, Aviv Chechik, Gal Fetaya, Ethan Machine Learning Machine unlearning aims to remove the influence of problematic training data after a model has been trained. The primary challenge in machine unlearning is ensuring that the process effectively removes specified data without compromising the model's overall performance on the remaining dataset. Many existing machine unlearning methods address this challenge by carefully balancing gradient ascent on the `unlearn' data with the gradient descent on a `retain' set that represents the training data. However, in many cases the training dataset is not fully available when we wish to unlearn some concepts, because models are released without their training datasets, and one may only have access to a $\textit{small part of a training set}$. Here, we propose OrthoGrad, a novel approach that mitigates interference between the unlearn set and a small retain set rather than competing ascent and descent processes. Our method projects the gradient of the unlearn set onto the subspace orthogonal to all gradients in the retain batch, effectively avoiding any gradient interference. We demonstrate the effectiveness of OrthoGrad on multiple machine unlearning benchmarks, including automatic speech recognition, outperforming competing methods. |
| title | Go Beyond Your Means: Unlearning with Per-Sample Gradient Orthogonalization |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2503.02312 |