Low-Rank Adapters Initialization via Gradient Surgery for Continual Learning
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2026
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866910214013321216 |
|---|---|
| author | Pasquali, Joana Barros, Ramiro N. Bianchessi, Arthur S. Turani, Vinícius Conte Abitante, João Vitor Boer Ravazio, Rafaela Cappelari Mattjie, Christian Parraga, Otávio Kupssinskü, Lucas S. Barros, Rodrigo C. |
| author_facet | Pasquali, Joana Barros, Ramiro N. Bianchessi, Arthur S. Turani, Vinícius Conte Abitante, João Vitor Boer Ravazio, Rafaela Cappelari Mattjie, Christian Parraga, Otávio Kupssinskü, Lucas S. Barros, Rodrigo C. |
| contents | LoRA is widely adopted for continual fine-tuning of Large Language Models due to its parameter efficiency, modularity across tasks, and compatibility with replay strategies. However, LoRA-based continual learning remains vulnerable to catastrophic forgetting, whose severity depends on how successive task gradients interact: when consecutive task gradients conflict, standard adapter initializations channel updates into subspaces that overwrite previously learned directions. We propose SLICE, a gradient-surgery-based initialization for LoRA adapters in continual learning. SLICE accumulates gradients from both the current task and a replay buffer of prior tasks, reconciles them through a projection operator, and decomposes the result via truncated SVD to initialize the adapter weights. We evaluate SLICE on the TRACE benchmark and sequences of Super-NI tasks, including a set of adversarial Super-NI sequences that we construct by mining task pairs with maximally opposing gradients. Compared to vanilla LoRA, LoRA-GA, and LoRAM, SLICE consistently achieves a better stability-plasticity trade-off, improving Average Performance, Final Performance and Forgetting metrics while preserving General Performance and In Context Performance across both standard and adversarial continual learning sequences. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_12752 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Low-Rank Adapters Initialization via Gradient Surgery for Continual Learning Pasquali, Joana Barros, Ramiro N. Bianchessi, Arthur S. Turani, Vinícius Conte Abitante, João Vitor Boer Ravazio, Rafaela Cappelari Mattjie, Christian Parraga, Otávio Kupssinskü, Lucas S. Barros, Rodrigo C. Machine Learning I.2.6 LoRA is widely adopted for continual fine-tuning of Large Language Models due to its parameter efficiency, modularity across tasks, and compatibility with replay strategies. However, LoRA-based continual learning remains vulnerable to catastrophic forgetting, whose severity depends on how successive task gradients interact: when consecutive task gradients conflict, standard adapter initializations channel updates into subspaces that overwrite previously learned directions. We propose SLICE, a gradient-surgery-based initialization for LoRA adapters in continual learning. SLICE accumulates gradients from both the current task and a replay buffer of prior tasks, reconciles them through a projection operator, and decomposes the result via truncated SVD to initialize the adapter weights. We evaluate SLICE on the TRACE benchmark and sequences of Super-NI tasks, including a set of adversarial Super-NI sequences that we construct by mining task pairs with maximally opposing gradients. Compared to vanilla LoRA, LoRA-GA, and LoRAM, SLICE consistently achieves a better stability-plasticity trade-off, improving Average Performance, Final Performance and Forgetting metrics while preserving General Performance and In Context Performance across both standard and adversarial continual learning sequences. |
| title | Low-Rank Adapters Initialization via Gradient Surgery for Continual Learning |
| topic | Machine Learning I.2.6 |
| url | https://arxiv.org/abs/2605.12752 |