FoRA: Fisher-orthogonal Rank Adaptation for Parameter-Efficient Fine-Tuning
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910271372525568 |
|---|---|
| author | Park, Juneyoung Lee, Seongbae Lee, Han-Sang Lee, Kyuho Kim, Minjae Hyeon, Seungheon Kwon, Kiduk Kim, Seongwan Lee, Jaeho |
| author_facet | Park, Juneyoung Lee, Seongbae Lee, Han-Sang Lee, Kyuho Kim, Minjae Hyeon, Seungheon Kwon, Kiduk Kim, Seongwan Lee, Jaeho |
| contents | Parameter-efficient fine-tuning(PEFT) has largely focused on LoRA and its accuracy-oriented variants, leaving the original goal of reducing trainable parameters has receivedcomparatively little attention. We introduce FoRA, which revisits this goal by reducing the number of adapted layers rather than adapter rank. FoRA selects task-informative layers via a single-pass diagonal Fisher score (under 1% of training cost) and trains the LoRA down-projection at selected layers on the Stiefel manifold, preserving column orthonormality and effective rank. FoRA consistently outperforms LoRA and DoRA at half their parameter budget, and falls within 0.7-0.8 accuracy points of AdaLoRA at one-quarter its parameter count, across five LLaMA-family backbones. Cross-architecture experiments on twelve backbones from the LLaMA, Qwen3, and Gemma families confirm consistent gains from 270M to 32B parameters. The two components combine super-additively: Fisher selection alone matches rank reduction at the same budget, while the Stiefel constraint provides the decisive additional gain. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_29317 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | FoRA: Fisher-orthogonal Rank Adaptation for Parameter-Efficient Fine-Tuning Park, Juneyoung Lee, Seongbae Lee, Han-Sang Lee, Kyuho Kim, Minjae Hyeon, Seungheon Kwon, Kiduk Kim, Seongwan Lee, Jaeho Computation and Language Parameter-efficient fine-tuning(PEFT) has largely focused on LoRA and its accuracy-oriented variants, leaving the original goal of reducing trainable parameters has receivedcomparatively little attention. We introduce FoRA, which revisits this goal by reducing the number of adapted layers rather than adapter rank. FoRA selects task-informative layers via a single-pass diagonal Fisher score (under 1% of training cost) and trains the LoRA down-projection at selected layers on the Stiefel manifold, preserving column orthonormality and effective rank. FoRA consistently outperforms LoRA and DoRA at half their parameter budget, and falls within 0.7-0.8 accuracy points of AdaLoRA at one-quarter its parameter count, across five LLaMA-family backbones. Cross-architecture experiments on twelve backbones from the LLaMA, Qwen3, and Gemma families confirm consistent gains from 270M to 32B parameters. The two components combine super-additively: Fisher selection alone matches rank reduction at the same budget, while the Stiefel constraint provides the decisive additional gain. |
| title | FoRA: Fisher-orthogonal Rank Adaptation for Parameter-Efficient Fine-Tuning |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2605.29317 |