Saved in:
| Main Authors: | , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2509.06100 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908782988099584 |
|---|---|
| author | Cao, Kefan Wu, Shuaicheng |
| author_facet | Cao, Kefan Wu, Shuaicheng |
| contents | Large language models (LLMs) suffer from catastrophic forgetting in sequential multi-task learning. Existing parameter regularization methods (e.g., O-LoRA, N-LoRA) mitigate interference via low-rank subspace orthogonality, but additive updates distort the intrinsic geometry of model parameters. We propose \textbf{OLieRA}, a Lie group based fine-tuning framework that preserves parameter geometry through multiplicative updates while enforcing orthogonality across task subspaces. OLieRA achieves state-of-the-art performance on the Standard CL benchmark and remains highly competitive under large task sequences. It further inherits the replay-free and task-ID free inference properties of O-LoRA, establishing a principled paradigm for continual learning in LLMs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_06100 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Orthogonal Low-rank Adaptation in Lie Groups for Continual Learning of Large Language Models Cao, Kefan Wu, Shuaicheng Computation and Language Large language models (LLMs) suffer from catastrophic forgetting in sequential multi-task learning. Existing parameter regularization methods (e.g., O-LoRA, N-LoRA) mitigate interference via low-rank subspace orthogonality, but additive updates distort the intrinsic geometry of model parameters. We propose \textbf{OLieRA}, a Lie group based fine-tuning framework that preserves parameter geometry through multiplicative updates while enforcing orthogonality across task subspaces. OLieRA achieves state-of-the-art performance on the Standard CL benchmark and remains highly competitive under large task sequences. It further inherits the replay-free and task-ID free inference properties of O-LoRA, establishing a principled paradigm for continual learning in LLMs. |
| title | Orthogonal Low-rank Adaptation in Lie Groups for Continual Learning of Large Language Models |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2509.06100 |