Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2601.18261 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911399206191104 |
|---|---|
| author | Tan, Chao-Hong Chen, Qian Wang, Wen Ma, Yukun Zhang, Chong Deng, Chong Zhang, Qinglin Li, Xiangang Ye, Jieping |
| author_facet | Tan, Chao-Hong Chen, Qian Wang, Wen Ma, Yukun Zhang, Chong Deng, Chong Zhang, Qinglin Li, Xiangang Ye, Jieping |
| contents | Catastrophic forgetting impairs the continuous learning of large language models. We propose Fisher-Guided Gradient Masking (FGGM), a framework that mitigates this by strategically selecting parameters for updates using diagonal Fisher Information. FGGM dynamically generates binary masks with adaptive thresholds, preserving critical parameters to balance stability and plasticity without requiring historical data. Unlike magnitude-based methods such as MIGU, our approach offers a mathematically principled parameter importance estimation. On the TRACE benchmark, FGGM shows a 9.6% relative improvement in retaining general capabilities over supervised fine-tuning (SFT) and a 4.4% improvement over MIGU on TRACE tasks. Additional analysis on code generation tasks confirms FGGM's superior performance and reduced forgetting, establishing it as an effective solution. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_18261 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | FGGM: Fisher-Guided Gradient Masking for Continual Learning Tan, Chao-Hong Chen, Qian Wang, Wen Ma, Yukun Zhang, Chong Deng, Chong Zhang, Qinglin Li, Xiangang Ye, Jieping Machine Learning Computation and Language Catastrophic forgetting impairs the continuous learning of large language models. We propose Fisher-Guided Gradient Masking (FGGM), a framework that mitigates this by strategically selecting parameters for updates using diagonal Fisher Information. FGGM dynamically generates binary masks with adaptive thresholds, preserving critical parameters to balance stability and plasticity without requiring historical data. Unlike magnitude-based methods such as MIGU, our approach offers a mathematically principled parameter importance estimation. On the TRACE benchmark, FGGM shows a 9.6% relative improvement in retaining general capabilities over supervised fine-tuning (SFT) and a 4.4% improvement over MIGU on TRACE tasks. Additional analysis on code generation tasks confirms FGGM's superior performance and reduced forgetting, establishing it as an effective solution. |
| title | FGGM: Fisher-Guided Gradient Masking for Continual Learning |
| topic | Machine Learning Computation and Language |
| url | https://arxiv.org/abs/2601.18261 |