Accurate and Efficient Low-Rank Model Merging in Core Space
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918315134287872 |
|---|---|
| author | Panariello, Aniello Marczak, Daniel Magistri, Simone Porrello, Angelo Twardowski, Bartłomiej Bagdanov, Andrew D. Calderara, Simone van de Weijer, Joost |
| author_facet | Panariello, Aniello Marczak, Daniel Magistri, Simone Porrello, Angelo Twardowski, Bartłomiej Bagdanov, Andrew D. Calderara, Simone van de Weijer, Joost |
| contents | In this paper, we address the challenges associated with merging low-rank adaptations of large neural networks. With the rise of parameter-efficient adaptation techniques, such as Low-Rank Adaptation (LoRA), model fine-tuning has become more accessible. While fine-tuning models with LoRA is highly efficient, existing merging methods often sacrifice this efficiency by merging fully-sized weight matrices. We propose the Core Space merging framework, which enables the merging of LoRA-adapted models within a common alignment basis, thereby preserving the efficiency of low-rank adaptation while substantially improving accuracy across tasks. We further provide a formal proof that projection into Core Space ensures no loss of information and provide a complexity analysis showing the efficiency gains. Extensive empirical results demonstrate that Core Space significantly improves existing merging techniques and achieves state-of-the-art results on both vision and language tasks while utilizing a fraction of the computational resources. Codebase is available at https://github.com/apanariello4/core-space-merging. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_17786 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Accurate and Efficient Low-Rank Model Merging in Core Space Panariello, Aniello Marczak, Daniel Magistri, Simone Porrello, Angelo Twardowski, Bartłomiej Bagdanov, Andrew D. Calderara, Simone van de Weijer, Joost Computer Vision and Pattern Recognition Artificial Intelligence In this paper, we address the challenges associated with merging low-rank adaptations of large neural networks. With the rise of parameter-efficient adaptation techniques, such as Low-Rank Adaptation (LoRA), model fine-tuning has become more accessible. While fine-tuning models with LoRA is highly efficient, existing merging methods often sacrifice this efficiency by merging fully-sized weight matrices. We propose the Core Space merging framework, which enables the merging of LoRA-adapted models within a common alignment basis, thereby preserving the efficiency of low-rank adaptation while substantially improving accuracy across tasks. We further provide a formal proof that projection into Core Space ensures no loss of information and provide a complexity analysis showing the efficiency gains. Extensive empirical results demonstrate that Core Space significantly improves existing merging techniques and achieves state-of-the-art results on both vision and language tasks while utilizing a fraction of the computational resources. Codebase is available at https://github.com/apanariello4/core-space-merging. |
| title | Accurate and Efficient Low-Rank Model Merging in Core Space |
| topic | Computer Vision and Pattern Recognition Artificial Intelligence |
| url | https://arxiv.org/abs/2509.17786 |