JumpLoRA: Sparse Adapters for Continual Learning in Large Language Models
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908998824886272 |
|---|---|
| author | Dragomir, Alexandra Pintilie, Ioana Barbalau, Antonio Dragoi, Marius Brad, Florin Paduraru, Cristian Daniel Tifrea, Alexandru Burceanu, Elena Ionescu, Radu Tudor |
| author_facet | Dragomir, Alexandra Pintilie, Ioana Barbalau, Antonio Dragoi, Marius Brad, Florin Paduraru, Cristian Daniel Tifrea, Alexandru Burceanu, Elena Ionescu, Radu Tudor |
| contents | Adapter-based methods have become a cost-effective approach to continual learning (CL) for Large Language Models (LLMs), by sequentially learning a low-rank update matrix for each task. To mitigate catastrophic forgetting, state-of-the-art approaches impose constraints on new adapters with respect to the previous ones, by targeting either subspace or coordinate-wise interference. In this paper, we propose JumpLoRA, a novel framework to adaptively induce sparsity in the Low-Rank Adaptation (LoRA) blocks through the use of JumpReLU gating. The method achieves dynamic parameter isolation, which helps prevent task interference. We demonstrate that our method is highly modular and compatible with LoRA-based CL approaches. Specifically, it significantly boosts the performance of IncLoRA and outperforms the leading state-of-the-art CL method, ELLA. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_16171 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | JumpLoRA: Sparse Adapters for Continual Learning in Large Language Models Dragomir, Alexandra Pintilie, Ioana Barbalau, Antonio Dragoi, Marius Brad, Florin Paduraru, Cristian Daniel Tifrea, Alexandru Burceanu, Elena Ionescu, Radu Tudor Machine Learning Artificial Intelligence Computation and Language Adapter-based methods have become a cost-effective approach to continual learning (CL) for Large Language Models (LLMs), by sequentially learning a low-rank update matrix for each task. To mitigate catastrophic forgetting, state-of-the-art approaches impose constraints on new adapters with respect to the previous ones, by targeting either subspace or coordinate-wise interference. In this paper, we propose JumpLoRA, a novel framework to adaptively induce sparsity in the Low-Rank Adaptation (LoRA) blocks through the use of JumpReLU gating. The method achieves dynamic parameter isolation, which helps prevent task interference. We demonstrate that our method is highly modular and compatible with LoRA-based CL approaches. Specifically, it significantly boosts the performance of IncLoRA and outperforms the leading state-of-the-art CL method, ELLA. |
| title | JumpLoRA: Sparse Adapters for Continual Learning in Large Language Models |
| topic | Machine Learning Artificial Intelligence Computation and Language |
| url | https://arxiv.org/abs/2604.16171 |