Parameter Importance is Not Static: Evolving Parameter Isolation for Supervised Fine-Tuning
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2026
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866914476006047744 |
|---|---|
| author | Lin, Zekai Xue, Chao Liang, Di Han, Xingsheng Liu, Peiyang Wu, Xianjie Jiang, Lei Lu, Yu Shi, Haibo Liang, Shuang Peng, Minlong |
| author_facet | Lin, Zekai Xue, Chao Liang, Di Han, Xingsheng Liu, Peiyang Wu, Xianjie Jiang, Lei Lu, Yu Shi, Haibo Liang, Shuang Peng, Minlong |
| contents | Supervised Fine-Tuning (SFT) of large language models often suffers from task interference and catastrophic forgetting. Recent approaches alleviate this issue by isolating task-critical parameters during training. However, these methods represent a static solution to a dynamic problem, assuming that parameter importance remains fixed once identified. In this work, we empirically demonstrate that parameter importance exhibits temporal drift over the course of training. To address this, we propose Evolving Parameter Isolation (EPI), a fine-tuning framework that adapts isolation decisions based on online estimates of parameter importance. Instead of freezing a fixed subset of parameters, EPI periodically updates isolation masks using gradient-based signals, enabling the model to protect emerging task-critical parameters while releasing outdated ones to recover plasticity. Experiments on diverse multi-task benchmarks demonstrate that EPI consistently reduces interference and forgetting compared to static isolation and standard fine-tuning, while improving overall generalization. Our analysis highlights the necessity of synchronizing isolation mechanisms with the evolving dynamics of learning diverse abilities. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_14010 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Parameter Importance is Not Static: Evolving Parameter Isolation for Supervised Fine-Tuning Lin, Zekai Xue, Chao Liang, Di Han, Xingsheng Liu, Peiyang Wu, Xianjie Jiang, Lei Lu, Yu Shi, Haibo Liang, Shuang Peng, Minlong Machine Learning Computation and Language Supervised Fine-Tuning (SFT) of large language models often suffers from task interference and catastrophic forgetting. Recent approaches alleviate this issue by isolating task-critical parameters during training. However, these methods represent a static solution to a dynamic problem, assuming that parameter importance remains fixed once identified. In this work, we empirically demonstrate that parameter importance exhibits temporal drift over the course of training. To address this, we propose Evolving Parameter Isolation (EPI), a fine-tuning framework that adapts isolation decisions based on online estimates of parameter importance. Instead of freezing a fixed subset of parameters, EPI periodically updates isolation masks using gradient-based signals, enabling the model to protect emerging task-critical parameters while releasing outdated ones to recover plasticity. Experiments on diverse multi-task benchmarks demonstrate that EPI consistently reduces interference and forgetting compared to static isolation and standard fine-tuning, while improving overall generalization. Our analysis highlights the necessity of synchronizing isolation mechanisms with the evolving dynamics of learning diverse abilities. |
| title | Parameter Importance is Not Static: Evolving Parameter Isolation for Supervised Fine-Tuning |
| topic | Machine Learning Computation and Language |
| url | https://arxiv.org/abs/2604.14010 |