LoRE-Merging: Exploring Low-Rank Estimation For Large Language Model Merging
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908375897341952 |
|---|---|
| author | Liu, Zehua Wu, Han Yao, Yuxuan She, Ruifeng Han, Xiongwei Zhong, Tao Yuan, Mingxuan |
| author_facet | Liu, Zehua Wu, Han Yao, Yuxuan She, Ruifeng Han, Xiongwei Zhong, Tao Yuan, Mingxuan |
| contents | While most current approaches rely on further training techniques, such as fine-tuning or reinforcement learning, to enhance model capacities, model merging stands out for its ability of improving models without requiring any additional training. In this paper, we propose a unified framework for model merging based on low-rank estimation of task vectors without the need for access to the base model, named \textsc{LoRE-Merging}. Our approach is motivated by the observation that task vectors from fine-tuned models frequently exhibit a limited number of dominant singular values, making low-rank estimations less prone to interference. We implement the method by formulating the merging problem as an optimization problem. Extensive empirical experiments demonstrate the effectiveness of our framework in mitigating interference and preserving task-specific information, thereby advancing the state-of-the-art performance in model merging techniques. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_10749 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | LoRE-Merging: Exploring Low-Rank Estimation For Large Language Model Merging Liu, Zehua Wu, Han Yao, Yuxuan She, Ruifeng Han, Xiongwei Zhong, Tao Yuan, Mingxuan Computation and Language Artificial Intelligence While most current approaches rely on further training techniques, such as fine-tuning or reinforcement learning, to enhance model capacities, model merging stands out for its ability of improving models without requiring any additional training. In this paper, we propose a unified framework for model merging based on low-rank estimation of task vectors without the need for access to the base model, named \textsc{LoRE-Merging}. Our approach is motivated by the observation that task vectors from fine-tuned models frequently exhibit a limited number of dominant singular values, making low-rank estimations less prone to interference. We implement the method by formulating the merging problem as an optimization problem. Extensive empirical experiments demonstrate the effectiveness of our framework in mitigating interference and preserving task-specific information, thereby advancing the state-of-the-art performance in model merging techniques. |
| title | LoRE-Merging: Exploring Low-Rank Estimation For Large Language Model Merging |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2502.10749 |