KeepLoRA: Continual Learning with Residual Gradient Adaptation
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908791595859968 |
|---|---|
| author | Luo, Mao-Lin Zhou, Zi-Hao Zhang, Yi-Lin Wan, Yuanyu Wei, Tong Zhang, Min-Ling |
| author_facet | Luo, Mao-Lin Zhou, Zi-Hao Zhang, Yi-Lin Wan, Yuanyu Wei, Tong Zhang, Min-Ling |
| contents | Continual learning for pre-trained vision-language models requires balancing three competing objectives: retaining pre-trained knowledge, preserving knowledge from a sequence of learned tasks, and maintaining the plasticity to acquire new knowledge. This paper presents a simple but effective approach called KeepLoRA to effectively balance these objectives. We first analyze the knowledge retention mechanism within the model parameter space and find that general knowledge is mainly encoded in the principal subspace, while task-specific knowledge is encoded in the residual subspace. Motivated by this finding, KeepLoRA learns new tasks by restricting LoRA parameter updates in the residual subspace to prevent interfering with previously learned capabilities. Specifically, we infuse knowledge for a new task by projecting its gradient onto a subspace orthogonal to both the principal subspace of pre-trained model and the dominant directions of previous task features. Our theoretical and empirical analyses confirm that KeepLoRA balances the three objectives and achieves state-of-the-art performance. The implementation code is available at https://github.com/MaolinLuo/KeepLoRA. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_19659 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | KeepLoRA: Continual Learning with Residual Gradient Adaptation Luo, Mao-Lin Zhou, Zi-Hao Zhang, Yi-Lin Wan, Yuanyu Wei, Tong Zhang, Min-Ling Computer Vision and Pattern Recognition Machine Learning Continual learning for pre-trained vision-language models requires balancing three competing objectives: retaining pre-trained knowledge, preserving knowledge from a sequence of learned tasks, and maintaining the plasticity to acquire new knowledge. This paper presents a simple but effective approach called KeepLoRA to effectively balance these objectives. We first analyze the knowledge retention mechanism within the model parameter space and find that general knowledge is mainly encoded in the principal subspace, while task-specific knowledge is encoded in the residual subspace. Motivated by this finding, KeepLoRA learns new tasks by restricting LoRA parameter updates in the residual subspace to prevent interfering with previously learned capabilities. Specifically, we infuse knowledge for a new task by projecting its gradient onto a subspace orthogonal to both the principal subspace of pre-trained model and the dominant directions of previous task features. Our theoretical and empirical analyses confirm that KeepLoRA balances the three objectives and achieves state-of-the-art performance. The implementation code is available at https://github.com/MaolinLuo/KeepLoRA. |
| title | KeepLoRA: Continual Learning with Residual Gradient Adaptation |
| topic | Computer Vision and Pattern Recognition Machine Learning |
| url | https://arxiv.org/abs/2601.19659 |