ReLearn: Unlearning via Learning for Large Language Models
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866910972101263360 |
|---|---|
| author | Xu, Haoming Zhao, Ningyuan Yang, Liming Zhao, Sendong Deng, Shumin Wang, Mengru Hooi, Bryan Oo, Nay Chen, Huajun Zhang, Ningyu |
| author_facet | Xu, Haoming Zhao, Ningyuan Yang, Liming Zhao, Sendong Deng, Shumin Wang, Mengru Hooi, Bryan Oo, Nay Chen, Huajun Zhang, Ningyu |
| contents | Current unlearning methods for large language models usually rely on reverse optimization to reduce target token probabilities. However, this paradigm disrupts the subsequent tokens prediction, degrading model performance and linguistic coherence. Moreover, existing evaluation metrics overemphasize contextual forgetting while inadequately assessing response fluency and relevance. To address these challenges, we propose ReLearn, a data augmentation and fine-tuning pipeline for effective unlearning, along with a comprehensive evaluation framework. This framework introduces Knowledge Forgetting Rate (KFR) and Knowledge Retention Rate (KRR) to measure knowledge-level preservation, and Linguistic Score (LS) to evaluate generation quality. Our experiments show that ReLearn successfully achieves targeted forgetting while preserving high-quality output. Through mechanistic analysis, we further demonstrate how reverse optimization disrupts coherent text generation, while ReLearn preserves this essential capability. Code is available at https://github.com/zjunlp/unlearn. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_11190 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | ReLearn: Unlearning via Learning for Large Language Models Xu, Haoming Zhao, Ningyuan Yang, Liming Zhao, Sendong Deng, Shumin Wang, Mengru Hooi, Bryan Oo, Nay Chen, Huajun Zhang, Ningyu Computation and Language Artificial Intelligence Computer Vision and Pattern Recognition Human-Computer Interaction Machine Learning Current unlearning methods for large language models usually rely on reverse optimization to reduce target token probabilities. However, this paradigm disrupts the subsequent tokens prediction, degrading model performance and linguistic coherence. Moreover, existing evaluation metrics overemphasize contextual forgetting while inadequately assessing response fluency and relevance. To address these challenges, we propose ReLearn, a data augmentation and fine-tuning pipeline for effective unlearning, along with a comprehensive evaluation framework. This framework introduces Knowledge Forgetting Rate (KFR) and Knowledge Retention Rate (KRR) to measure knowledge-level preservation, and Linguistic Score (LS) to evaluate generation quality. Our experiments show that ReLearn successfully achieves targeted forgetting while preserving high-quality output. Through mechanistic analysis, we further demonstrate how reverse optimization disrupts coherent text generation, while ReLearn preserves this essential capability. Code is available at https://github.com/zjunlp/unlearn. |
| title | ReLearn: Unlearning via Learning for Large Language Models |
| topic | Computation and Language Artificial Intelligence Computer Vision and Pattern Recognition Human-Computer Interaction Machine Learning |
| url | https://arxiv.org/abs/2502.11190 |