Not All Directions Matter: Towards Structured and Task-Aware Low-Rank Model Adaptation
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918467322511360 |
|---|---|
| author | Xiao, Xi Ma, Chenrui Zhang, Yunbei Liu, Chen Wang, Zhuxuanzi Li, Yanshu Zhao, Lin Hu, Guosheng Wang, Tianyang Xu, Hao |
| author_facet | Xiao, Xi Ma, Chenrui Zhang, Yunbei Liu, Chen Wang, Zhuxuanzi Li, Yanshu Zhao, Lin Hu, Guosheng Wang, Tianyang Xu, Hao |
| contents | Low-Rank Adaptation (LoRA) has become a cornerstone of parameter-efficient fine-tuning (PEFT). Yet, its efficacy is hampered by two fundamental limitations: semantic drift, by treating all update directions with equal importance, and structural incoherence, from adapting layers independently, resulting in suboptimal, uncoordinated updates. To remedy these, we propose StructLoRA, a framework that addresses both limitations through a principled, dual-component design: (1) an Information Bottleneck-guided filter that prunes task-irrelevant directions to mitigate semantic drift, and (2) a lightweight, training-only graph-based coordinator that enforces inter-layer consistency to resolve structural incoherence. Extensive experiments across large language model , vision language model, and vision model (including LLaMA, LLaVA, and ViT) demonstrate that StructLoRA consistently establishes a new state-of-the-art, outperforming not only vanilla LoRA but also advanced dynamic rank allocation and sparsity-based methods. Notably, the benefits are particularly pronounced in challenging low-rank and low-data regimes. Crucially, since our proposed modules operate only during training, StructLoRA enhances performance with zero additional inference cost, advancing the focus of PEFT -- from mere parameter compression to a more holistic optimization of information quality and structural integrity. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_14228 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Not All Directions Matter: Towards Structured and Task-Aware Low-Rank Model Adaptation Xiao, Xi Ma, Chenrui Zhang, Yunbei Liu, Chen Wang, Zhuxuanzi Li, Yanshu Zhao, Lin Hu, Guosheng Wang, Tianyang Xu, Hao Computer Vision and Pattern Recognition Low-Rank Adaptation (LoRA) has become a cornerstone of parameter-efficient fine-tuning (PEFT). Yet, its efficacy is hampered by two fundamental limitations: semantic drift, by treating all update directions with equal importance, and structural incoherence, from adapting layers independently, resulting in suboptimal, uncoordinated updates. To remedy these, we propose StructLoRA, a framework that addresses both limitations through a principled, dual-component design: (1) an Information Bottleneck-guided filter that prunes task-irrelevant directions to mitigate semantic drift, and (2) a lightweight, training-only graph-based coordinator that enforces inter-layer consistency to resolve structural incoherence. Extensive experiments across large language model , vision language model, and vision model (including LLaMA, LLaVA, and ViT) demonstrate that StructLoRA consistently establishes a new state-of-the-art, outperforming not only vanilla LoRA but also advanced dynamic rank allocation and sparsity-based methods. Notably, the benefits are particularly pronounced in challenging low-rank and low-data regimes. Crucially, since our proposed modules operate only during training, StructLoRA enhances performance with zero additional inference cost, advancing the focus of PEFT -- from mere parameter compression to a more holistic optimization of information quality and structural integrity. |
| title | Not All Directions Matter: Towards Structured and Task-Aware Low-Rank Model Adaptation |
| topic | Computer Vision and Pattern Recognition |
| url | https://arxiv.org/abs/2603.14228 |