FroM: Frobenius Norm-Based Data-Free Adaptive Model Merging
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918142808162304 |
|---|---|
| author | Li, Zijian Feng, Xiaocheng Liu, Huixin Huang, Yichong Liu, Ting Qin, Bing |
| author_facet | Li, Zijian Feng, Xiaocheng Liu, Huixin Huang, Yichong Liu, Ting Qin, Bing |
| contents | With the development of large language models, fine-tuning has emerged as an effective method to enhance performance in specific scenarios by injecting domain-specific knowledge. In this context, model merging techniques provide a solution for fusing knowledge from multiple fine-tuning models by combining their parameters. However, traditional methods often encounter task interference when merging full fine-tuning models, and this problem becomes even more evident in parameter-efficient fine-tuning scenarios. In this paper, we introduce an improvement to the RegMean method, which indirectly leverages the training data to approximate the outputs of the linear layers before and after merging. We propose an adaptive merging method called FroM, which directly measures the model parameters using the Frobenius norm, without any training data. By introducing an additional hyperparameter for control, FroM outperforms baseline methods across various fine-tuning scenarios, alleviating the task interference problem. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_02478 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | FroM: Frobenius Norm-Based Data-Free Adaptive Model Merging Li, Zijian Feng, Xiaocheng Liu, Huixin Huang, Yichong Liu, Ting Qin, Bing Computation and Language With the development of large language models, fine-tuning has emerged as an effective method to enhance performance in specific scenarios by injecting domain-specific knowledge. In this context, model merging techniques provide a solution for fusing knowledge from multiple fine-tuning models by combining their parameters. However, traditional methods often encounter task interference when merging full fine-tuning models, and this problem becomes even more evident in parameter-efficient fine-tuning scenarios. In this paper, we introduce an improvement to the RegMean method, which indirectly leverages the training data to approximate the outputs of the linear layers before and after merging. We propose an adaptive merging method called FroM, which directly measures the model parameters using the Frobenius norm, without any training data. By introducing an additional hyperparameter for control, FroM outperforms baseline methods across various fine-tuning scenarios, alleviating the task interference problem. |
| title | FroM: Frobenius Norm-Based Data-Free Adaptive Model Merging |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2506.02478 |