Sparse Memory Finetuning as a Low-Forgetting Alternative to LoRA and Full Finetuning
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911646578900992 |
|---|---|
| author | Gupta, Prakhar Shah, Garv Goyal, Satyam Kanchi, Anirudh |
| author_facet | Gupta, Prakhar Shah, Garv Goyal, Satyam Kanchi, Anirudh |
| contents | Adapting a pretrained language model to a new task often hurts the general capabilities it already had, a problem known as catastrophic forgetting. Sparse Memory Finetuning (SMF) tries to avoid this by adding key-value memory layers to the model and, on each training step, updating only the small set of memory rows that the current batch reads most heavily. We re-implement SMF on Qwen-2.5-0.5B-Instruct and compare it with LoRA and full finetuning on MedMCQA, a 4-choice medical exam task, using WikiText perplexity and TriviaQA accuracy as forgetting probes. SMF improves MedMCQA by 2.5 percentage points while keeping both forgetting probes within roughly 1 point of the base model, whereas LoRA and full finetuning achieve larger gains but with clear drift on both. We also compare two row-selection rules (KL-divergence and TF-IDF), which balance the two forgetting metrics differently. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_03229 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Sparse Memory Finetuning as a Low-Forgetting Alternative to LoRA and Full Finetuning Gupta, Prakhar Shah, Garv Goyal, Satyam Kanchi, Anirudh Computation and Language Machine Learning Adapting a pretrained language model to a new task often hurts the general capabilities it already had, a problem known as catastrophic forgetting. Sparse Memory Finetuning (SMF) tries to avoid this by adding key-value memory layers to the model and, on each training step, updating only the small set of memory rows that the current batch reads most heavily. We re-implement SMF on Qwen-2.5-0.5B-Instruct and compare it with LoRA and full finetuning on MedMCQA, a 4-choice medical exam task, using WikiText perplexity and TriviaQA accuracy as forgetting probes. SMF improves MedMCQA by 2.5 percentage points while keeping both forgetting probes within roughly 1 point of the base model, whereas LoRA and full finetuning achieve larger gains but with clear drift on both. We also compare two row-selection rules (KL-divergence and TF-IDF), which balance the two forgetting metrics differently. |
| title | Sparse Memory Finetuning as a Low-Forgetting Alternative to LoRA and Full Finetuning |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2605.03229 |