Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2512.00878 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912773955387392 |
|---|---|
| author | Tian, Chunlin Wei, Xuyang Liu, Huanrong Guo, Zhijiang Li, Li |
| author_facet | Tian, Chunlin Wei, Xuyang Liu, Huanrong Guo, Zhijiang Li, Li |
| contents | Low-Rank Adaptation (LoRA) is a widely adopted parameter-efficient fine-tuning (PEFT) method for Large Language Models (LLMs), but it still incurs notable overhead and suffers from parameter interference in complex datasets. While recent works decouple LoRA update matrices to exploit matrix-wise asymmetry, training costs remain high. We revisit LoRA from the perspective of inter-matrix and intra-layer parameter redundancy and propose Resource-Efficient Low-Rank Adaptation, EffiLoRA, a lightweight and generalizable approach for language, multimodal, and diffusion models. EffiLoRA employs a unified A matrix across all transformer layers and introduces a runtime selective B matrices update to dynamically trade-off the system resource budget and model performance. EffiLoRA consistently outperforms LoRA across diverse modalities, including commonsense reasoning, visual instruction tuning, and image generation, demonstrating improved efficiency and robustness. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_00878 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Less is More: Resource-Efficient Low-Rank Adaptation Tian, Chunlin Wei, Xuyang Liu, Huanrong Guo, Zhijiang Li, Li Computation and Language Artificial Intelligence Low-Rank Adaptation (LoRA) is a widely adopted parameter-efficient fine-tuning (PEFT) method for Large Language Models (LLMs), but it still incurs notable overhead and suffers from parameter interference in complex datasets. While recent works decouple LoRA update matrices to exploit matrix-wise asymmetry, training costs remain high. We revisit LoRA from the perspective of inter-matrix and intra-layer parameter redundancy and propose Resource-Efficient Low-Rank Adaptation, EffiLoRA, a lightweight and generalizable approach for language, multimodal, and diffusion models. EffiLoRA employs a unified A matrix across all transformer layers and introduces a runtime selective B matrices update to dynamically trade-off the system resource budget and model performance. EffiLoRA consistently outperforms LoRA across diverse modalities, including commonsense reasoning, visual instruction tuning, and image generation, demonstrating improved efficiency and robustness. |
| title | Less is More: Resource-Efficient Low-Rank Adaptation |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2512.00878 |