Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2509.09119 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915489332068352 |
|---|---|
| author | Zhang, Hao Huang, Bo Li, Zhenjia Xiao, Xi Leong, Hui Yi Zhang, Zumeng Long, Xinwei Wang, Tianyang Xu, Hao |
| author_facet | Zhang, Hao Huang, Bo Li, Zhenjia Xiao, Xi Leong, Hui Yi Zhang, Zumeng Long, Xinwei Wang, Tianyang Xu, Hao |
| contents | Large Language Models (LLMs) have transformed both everyday life and scientific research. However, adapting LLMs from general-purpose models to specialized tasks remains challenging, particularly in resource-constrained environments. Low-Rank Adaptation (LoRA), a prominent method within Parameter-Efficient Fine-Tuning (PEFT), has emerged as a promising approach to LLMs by approximating model weight updates using low-rank decomposition. However, LoRA is limited by its uniform rank ( r ) allocation to each incremental matrix, and existing rank allocation techniques aimed at addressing this issue remain computationally inefficient, complex, and unstable, hindering practical applications. To address these limitations, we propose Sensitivity-LoRA, an efficient fine-tuning method that dynamically allocates ranks to weight matrices based on both their global and local sensitivities. It leverages the second-order derivatives (Hessian Matrix) of the loss function to effectively capture weight sensitivity, enabling optimal rank allocation with minimal computational overhead. Our experimental results have demonstrated robust effectiveness, efficiency and stability of Sensitivity-LoRA across diverse tasks and benchmarks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_09119 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Sensitivity-LoRA: Low-Load Sensitivity-Based Fine-Tuning for Large Language Models Zhang, Hao Huang, Bo Li, Zhenjia Xiao, Xi Leong, Hui Yi Zhang, Zumeng Long, Xinwei Wang, Tianyang Xu, Hao Machine Learning Large Language Models (LLMs) have transformed both everyday life and scientific research. However, adapting LLMs from general-purpose models to specialized tasks remains challenging, particularly in resource-constrained environments. Low-Rank Adaptation (LoRA), a prominent method within Parameter-Efficient Fine-Tuning (PEFT), has emerged as a promising approach to LLMs by approximating model weight updates using low-rank decomposition. However, LoRA is limited by its uniform rank ( r ) allocation to each incremental matrix, and existing rank allocation techniques aimed at addressing this issue remain computationally inefficient, complex, and unstable, hindering practical applications. To address these limitations, we propose Sensitivity-LoRA, an efficient fine-tuning method that dynamically allocates ranks to weight matrices based on both their global and local sensitivities. It leverages the second-order derivatives (Hessian Matrix) of the loss function to effectively capture weight sensitivity, enabling optimal rank allocation with minimal computational overhead. Our experimental results have demonstrated robust effectiveness, efficiency and stability of Sensitivity-LoRA across diverse tasks and benchmarks. |
| title | Sensitivity-LoRA: Low-Load Sensitivity-Based Fine-Tuning for Large Language Models |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2509.09119 |