Will it Merge? On The Causes of Model Mergeability
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908757071495168 |
|---|---|
| author | Rahamim, Adir Yehudai, Asaf Carmeli, Boaz Choshen, Leshem Mass, Yosi Belinkov, Yonatan |
| author_facet | Rahamim, Adir Yehudai, Asaf Carmeli, Boaz Choshen, Leshem Mass, Yosi Belinkov, Yonatan |
| contents | Model merging has emerged as a promising technique for combining multiple fine-tuned models into a single multitask model without retraining. However, the factors that determine whether merging will succeed or fail remain poorly understood. In this work, we investigate why specific models are merged better than others. To do so, we propose a concrete, measurable definition of mergeability. We investigate several potential causes for high or low mergeability, highlighting the base model knowledge as a dominant factor: Models fine-tuned on instances that the base model knows better are more mergeable than models fine-tuned on instances that the base model struggles with. Based on our mergeability definition, we explore a simple weighted merging technique that better preserves weak knowledge in the base model. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_06672 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Will it Merge? On The Causes of Model Mergeability Rahamim, Adir Yehudai, Asaf Carmeli, Boaz Choshen, Leshem Mass, Yosi Belinkov, Yonatan Computation and Language Model merging has emerged as a promising technique for combining multiple fine-tuned models into a single multitask model without retraining. However, the factors that determine whether merging will succeed or fail remain poorly understood. In this work, we investigate why specific models are merged better than others. To do so, we propose a concrete, measurable definition of mergeability. We investigate several potential causes for high or low mergeability, highlighting the base model knowledge as a dominant factor: Models fine-tuned on instances that the base model knows better are more mergeable than models fine-tuned on instances that the base model struggles with. Based on our mergeability definition, we explore a simple weighted merging technique that better preserves weak knowledge in the base model. |
| title | Will it Merge? On The Causes of Model Mergeability |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2601.06672 |