On the Limits of Model Merging for Multilinguality in Pre-Training
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2026
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866917531654029312 |
|---|---|
| author | Aycock, Seth Vitiugin, Fedor Umnov, Aleksandr Monz, Christof Sima'an, Khalil |
| author_facet | Aycock, Seth Vitiugin, Fedor Umnov, Aleksandr Monz, Christof Sima'an, Khalil |
| contents | Endowing models with consistent multilingual performance can be achieved by mixing pre-training data, or post-training approaches such as language-specific model merging. In this work, we test whether merging can be applied to monolingually pre-trained models. We conduct a controlled study on the efficacy of mixed, merged, and monolingual pre-training setups. We find that while monolingual pre-training results in strong in-language performance, merging any combination of monolingual models leads to performance collapse due to interference. Our analysis suggests representational similarity is a prerequisite for model merging. We therefore conclude that the flexibility of merging in fine-tuning does not extend trivially to language-specific pre-training. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_25846 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | On the Limits of Model Merging for Multilinguality in Pre-Training Aycock, Seth Vitiugin, Fedor Umnov, Aleksandr Monz, Christof Sima'an, Khalil Computation and Language Endowing models with consistent multilingual performance can be achieved by mixing pre-training data, or post-training approaches such as language-specific model merging. In this work, we test whether merging can be applied to monolingually pre-trained models. We conduct a controlled study on the efficacy of mixed, merged, and monolingual pre-training setups. We find that while monolingual pre-training results in strong in-language performance, merging any combination of monolingual models leads to performance collapse due to interference. Our analysis suggests representational similarity is a prerequisite for model merging. We therefore conclude that the flexibility of merging in fine-tuning does not extend trivially to language-specific pre-training. |
| title | On the Limits of Model Merging for Multilinguality in Pre-Training |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2605.25846 |