Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2505.22120 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908670209556480 |
|---|---|
| author | Wang, Runyu Ping, Peng Guo, Zhengyu Zhang, Xiaoye Shi, Quan Zhou, Liting Ji, Tianbo |
| author_facet | Wang, Runyu Ping, Peng Guo, Zhengyu Zhang, Xiaoye Shi, Quan Zhou, Liting Ji, Tianbo |
| contents | Fine-tuning adapts pretrained models for specific tasks but poses the risk of catastrophic forgetting (CF), where critical knowledge from pretraining is overwritten. To address the issue of CF in a general-purpose framework, we propose Low-damage Knowledge Implanting (LoKI), a parameter-efficient fine-tuning (PEFT) technique that utilizes recent mechanistic understanding of how knowledge is stored in transformer architectures. We compare LoKI against state-of-the-art PEFT methods in two real-world fine-tuning scenarios. The results show that LoKI demonstrates significantly better preservation of general capabilities. At the same time, its task-specific performance is comparable to or even surpasses that of full parameter fine-tuning and these PEFT methods across various model architectures. Our work bridges the mechanistic insights of LLMs' knowledge storage with practical fine-tuning objectives, enabling an effective balance between task-specific adaptation and the retention of general-purpose capabilities. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_22120 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | LoKI: Low-damage Knowledge Implanting of Large Language Models Wang, Runyu Ping, Peng Guo, Zhengyu Zhang, Xiaoye Shi, Quan Zhou, Liting Ji, Tianbo Computation and Language Fine-tuning adapts pretrained models for specific tasks but poses the risk of catastrophic forgetting (CF), where critical knowledge from pretraining is overwritten. To address the issue of CF in a general-purpose framework, we propose Low-damage Knowledge Implanting (LoKI), a parameter-efficient fine-tuning (PEFT) technique that utilizes recent mechanistic understanding of how knowledge is stored in transformer architectures. We compare LoKI against state-of-the-art PEFT methods in two real-world fine-tuning scenarios. The results show that LoKI demonstrates significantly better preservation of general capabilities. At the same time, its task-specific performance is comparable to or even surpasses that of full parameter fine-tuning and these PEFT methods across various model architectures. Our work bridges the mechanistic insights of LLMs' knowledge storage with practical fine-tuning objectives, enabling an effective balance between task-specific adaptation and the retention of general-purpose capabilities. |
| title | LoKI: Low-damage Knowledge Implanting of Large Language Models |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2505.22120 |