$α$-LoRA: Effective Fine-Tuning via Base Model Rescaling
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909867685445632 |
|---|---|
| author | Firdoussi, Aymane El Chayti, El Mahdi Seddik, Mohamed El Amine Jaggi, Martin |
| author_facet | Firdoussi, Aymane El Chayti, El Mahdi Seddik, Mohamed El Amine Jaggi, Martin |
| contents | Fine-tuning has proven to be highly effective in adapting pre-trained models to perform better on new desired tasks with minimal data samples. Among the most widely used approaches are reparameterization methods, which update a target module by augmenting its frozen weight matrix with an additional trainable weight matrix. The most prominent example is Low Rank Adaption (LoRA), which gained significant attention in recent years. In this paper, we introduce a new class of reparameterization methods for transfer learning, designed to enhance the generalization ability of fine-tuned models. We establish the effectiveness of our approach in a high-dimensional binary classification setting using tools from Random Matrix Theory, and further validate our theoretical findings through more realistic experiments, such as fine-tuning LLMs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_21345 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | $α$-LoRA: Effective Fine-Tuning via Base Model Rescaling Firdoussi, Aymane El Chayti, El Mahdi Seddik, Mohamed El Amine Jaggi, Martin Machine Learning Artificial Intelligence Fine-tuning has proven to be highly effective in adapting pre-trained models to perform better on new desired tasks with minimal data samples. Among the most widely used approaches are reparameterization methods, which update a target module by augmenting its frozen weight matrix with an additional trainable weight matrix. The most prominent example is Low Rank Adaption (LoRA), which gained significant attention in recent years. In this paper, we introduce a new class of reparameterization methods for transfer learning, designed to enhance the generalization ability of fine-tuned models. We establish the effectiveness of our approach in a high-dimensional binary classification setting using tools from Random Matrix Theory, and further validate our theoretical findings through more realistic experiments, such as fine-tuning LLMs. |
| title | $α$-LoRA: Effective Fine-Tuning via Base Model Rescaling |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2510.21345 |