LoRS: Efficient Low-Rank Adaptation for Sparse Large Language Model
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909457002266624 |
|---|---|
| author | Hu, Yuxuan Zhang, Jing Chen, Xiaodong Zhao, Zhe Li, Cuiping Chen, Hong |
| author_facet | Hu, Yuxuan Zhang, Jing Chen, Xiaodong Zhao, Zhe Li, Cuiping Chen, Hong |
| contents | Existing low-rank adaptation (LoRA) methods face challenges on sparse large language models (LLMs) due to the inability to maintain sparsity. Recent works introduced methods that maintain sparsity by augmenting LoRA techniques with additional masking mechanisms. Despite these successes, such approaches suffer from an increased memory and computation overhead, which affects efficiency of LoRA methods. In response to this limitation, we introduce LoRS, an innovative method designed to achieve both memory and computation efficiency when fine-tuning sparse LLMs. To mitigate the substantial memory and computation demands associated with preserving sparsity, our approach incorporates strategies of weight recompute and computational graph rearrangement. In addition, we also improve the effectiveness of LoRS through better adapter initialization. These innovations lead to a notable reduction in memory and computation consumption during the fine-tuning phase, all while achieving performance levels that outperform existing LoRA approaches. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_08582 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | LoRS: Efficient Low-Rank Adaptation for Sparse Large Language Model Hu, Yuxuan Zhang, Jing Chen, Xiaodong Zhao, Zhe Li, Cuiping Chen, Hong Computation and Language I.2.7 Existing low-rank adaptation (LoRA) methods face challenges on sparse large language models (LLMs) due to the inability to maintain sparsity. Recent works introduced methods that maintain sparsity by augmenting LoRA techniques with additional masking mechanisms. Despite these successes, such approaches suffer from an increased memory and computation overhead, which affects efficiency of LoRA methods. In response to this limitation, we introduce LoRS, an innovative method designed to achieve both memory and computation efficiency when fine-tuning sparse LLMs. To mitigate the substantial memory and computation demands associated with preserving sparsity, our approach incorporates strategies of weight recompute and computational graph rearrangement. In addition, we also improve the effectiveness of LoRS through better adapter initialization. These innovations lead to a notable reduction in memory and computation consumption during the fine-tuning phase, all while achieving performance levels that outperform existing LoRA approaches. |
| title | LoRS: Efficient Low-Rank Adaptation for Sparse Large Language Model |
| topic | Computation and Language I.2.7 |
| url | https://arxiv.org/abs/2501.08582 |