RoRA: Efficient Fine-Tuning of LLM with Reliability Optimization for Rank Adaptation
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912183999266816 |
|---|---|
| author | Liu, Jun Kong, Zhenglun Dong, Peiyan Yang, Changdi Shen, Xuan Zhao, Pu Tang, Hao Yuan, Geng Niu, Wei Zhang, Wenbin Lin, Xue Huang, Dong Wang, Yanzhi |
| author_facet | Liu, Jun Kong, Zhenglun Dong, Peiyan Yang, Changdi Shen, Xuan Zhao, Pu Tang, Hao Yuan, Geng Niu, Wei Zhang, Wenbin Lin, Xue Huang, Dong Wang, Yanzhi |
| contents | Fine-tuning helps large language models (LLM) recover degraded information and enhance task performance. Although Low-Rank Adaptation (LoRA) is widely used and effective for fine-tuning, we have observed that its scaling factor can limit or even reduce performance as the rank size increases. To address this issue, we propose RoRA (Rank-adaptive Reliability Optimization), a simple yet effective method for optimizing LoRA's scaling factor. By replacing $α/r$ with $α/\sqrt{r}$, RoRA ensures improved performance as rank size increases. Moreover, RoRA enhances low-rank adaptation in fine-tuning uncompressed models and excels in the more challenging task of accuracy recovery when fine-tuning pruned models. Extensive experiments demonstrate the effectiveness of RoRA in fine-tuning both uncompressed and pruned models. RoRA surpasses the state-of-the-art (SOTA) in average accuracy and robustness on LLaMA-7B/13B, LLaMA2-7B, and LLaMA3-8B, specifically outperforming LoRA and DoRA by 6.5% and 2.9% on LLaMA-7B, respectively. In pruned model fine-tuning, RoRA shows significant advantages; for SHEARED-LLAMA-1.3, a LLaMA-7B with 81.4% pruning, RoRA achieves 5.7% higher average accuracy than LoRA and 3.9% higher than DoRA. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_04315 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | RoRA: Efficient Fine-Tuning of LLM with Reliability Optimization for Rank Adaptation Liu, Jun Kong, Zhenglun Dong, Peiyan Yang, Changdi Shen, Xuan Zhao, Pu Tang, Hao Yuan, Geng Niu, Wei Zhang, Wenbin Lin, Xue Huang, Dong Wang, Yanzhi Machine Learning Artificial Intelligence Fine-tuning helps large language models (LLM) recover degraded information and enhance task performance. Although Low-Rank Adaptation (LoRA) is widely used and effective for fine-tuning, we have observed that its scaling factor can limit or even reduce performance as the rank size increases. To address this issue, we propose RoRA (Rank-adaptive Reliability Optimization), a simple yet effective method for optimizing LoRA's scaling factor. By replacing $α/r$ with $α/\sqrt{r}$, RoRA ensures improved performance as rank size increases. Moreover, RoRA enhances low-rank adaptation in fine-tuning uncompressed models and excels in the more challenging task of accuracy recovery when fine-tuning pruned models. Extensive experiments demonstrate the effectiveness of RoRA in fine-tuning both uncompressed and pruned models. RoRA surpasses the state-of-the-art (SOTA) in average accuracy and robustness on LLaMA-7B/13B, LLaMA2-7B, and LLaMA3-8B, specifically outperforming LoRA and DoRA by 6.5% and 2.9% on LLaMA-7B, respectively. In pruned model fine-tuning, RoRA shows significant advantages; for SHEARED-LLAMA-1.3, a LLaMA-7B with 81.4% pruning, RoRA achieves 5.7% higher average accuracy than LoRA and 3.9% higher than DoRA. |
| title | RoRA: Efficient Fine-Tuning of LLM with Reliability Optimization for Rank Adaptation |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2501.04315 |