NAN: A Training-Free Solution to Coefficient Estimation in Model Merging
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866912386792816640 |
|---|---|
| author | Si, Chongjie Lv, Kangtao Jiang, Jingjing Wang, Yadao Wang, Yongwei Yang, Xiaokang Su, Wenbo Zheng, Bo Shen, Wei |
| author_facet | Si, Chongjie Lv, Kangtao Jiang, Jingjing Wang, Yadao Wang, Yongwei Yang, Xiaokang Su, Wenbo Zheng, Bo Shen, Wei |
| contents | Model merging offers a training-free alternative to multi-task learning by combining independently fine-tuned models into a unified one without access to raw data. However, existing approaches often rely on heuristics to determine the merging coefficients, limiting their scalability and generality. In this work, we revisit model merging through the lens of least-squares optimization and show that the optimal merging weights should scale with the amount of task-specific information encoded in each model. Based on this insight, we propose NAN, a simple yet effective method that estimates model merging coefficients via the inverse of parameter norm. NAN is training-free, plug-and-play, and applicable to a wide range of merging strategies. Extensive experiments on show that NAN consistently improves performance of baseline methods. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_16148 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | NAN: A Training-Free Solution to Coefficient Estimation in Model Merging Si, Chongjie Lv, Kangtao Jiang, Jingjing Wang, Yadao Wang, Yongwei Yang, Xiaokang Su, Wenbo Zheng, Bo Shen, Wei Machine Learning Computation and Language Model merging offers a training-free alternative to multi-task learning by combining independently fine-tuned models into a unified one without access to raw data. However, existing approaches often rely on heuristics to determine the merging coefficients, limiting their scalability and generality. In this work, we revisit model merging through the lens of least-squares optimization and show that the optimal merging weights should scale with the amount of task-specific information encoded in each model. Based on this insight, we propose NAN, a simple yet effective method that estimates model merging coefficients via the inverse of parameter norm. NAN is training-free, plug-and-play, and applicable to a wide range of merging strategies. Extensive experiments on show that NAN consistently improves performance of baseline methods. |
| title | NAN: A Training-Free Solution to Coefficient Estimation in Model Merging |
| topic | Machine Learning Computation and Language |
| url | https://arxiv.org/abs/2505.16148 |