Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2512.23485 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914223850782720 |
|---|---|
| author | Wan, Guoan Chen, Tianyu Feng, Fangzheng Zhou, Haoyi Xu, Runhua |
| author_facet | Wan, Guoan Chen, Tianyu Feng, Fangzheng Zhou, Haoyi Xu, Runhua |
| contents | Parameter-efficient fine-tuning (PEFT) methods have emerged as a practical solution for adapting large foundation models to downstream tasks, reducing computational and memory costs by updating only a small subset of parameters. Among them, approaches like LoRA aim to strike a balance between efficiency and expressiveness, but often suffer from slow convergence and limited adaptation capacity due to their inherent low-rank constraints. This trade-off hampers the ability of PEFT methods to capture complex patterns needed for diverse tasks. To address these challenges, we propose FRoD, a novel fine-tuning method that combines hierarchical joint decomposition with rotational degrees of freedom. By extracting a globally shared basis across layers and injecting sparse, learnable perturbations into scaling factors for flexible full-rank updates, FRoD enhances expressiveness and efficiency, leading to faster and more robust convergence. On 20 benchmarks spanning vision, reasoning, and language understanding, FRoD matches full model fine-tuning in accuracy, while using only 1.72% of trainable parameters under identical training budgets. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_23485 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | FRoD: Full-Rank Efficient Fine-Tuning with Rotational Degrees for Fast Convergence Wan, Guoan Chen, Tianyu Feng, Fangzheng Zhou, Haoyi Xu, Runhua Machine Learning Artificial Intelligence Parameter-efficient fine-tuning (PEFT) methods have emerged as a practical solution for adapting large foundation models to downstream tasks, reducing computational and memory costs by updating only a small subset of parameters. Among them, approaches like LoRA aim to strike a balance between efficiency and expressiveness, but often suffer from slow convergence and limited adaptation capacity due to their inherent low-rank constraints. This trade-off hampers the ability of PEFT methods to capture complex patterns needed for diverse tasks. To address these challenges, we propose FRoD, a novel fine-tuning method that combines hierarchical joint decomposition with rotational degrees of freedom. By extracting a globally shared basis across layers and injecting sparse, learnable perturbations into scaling factors for flexible full-rank updates, FRoD enhances expressiveness and efficiency, leading to faster and more robust convergence. On 20 benchmarks spanning vision, reasoning, and language understanding, FRoD matches full model fine-tuning in accuracy, while using only 1.72% of trainable parameters under identical training budgets. |
| title | FRoD: Full-Rank Efficient Fine-Tuning with Rotational Degrees for Fast Convergence |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2512.23485 |