AlignFreeze: Navigating the Impact of Realignment on the Layers of Multilingual Models Across Diverse Languages
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866910834749341696 |
|---|---|
| author | Bakos, Steve Gaschi, Félix Guzmán, David More, Riddhi Li, Kelly Chutong Lee, En-Shiun Annie |
| author_facet | Bakos, Steve Gaschi, Félix Guzmán, David More, Riddhi Li, Kelly Chutong Lee, En-Shiun Annie |
| contents | Realignment techniques are often employed to enhance cross-lingual transfer in multilingual language models, still, they can sometimes degrade performance in languages that differ significantly from the fine-tuned source language. This paper introduces AlignFreeze, a method that freezes either the layers' lower half or upper half during realignment. Through controlled experiments on 4 tasks, 3 models, and in 35 languages, we find that realignment affects all the layers but can be the most detrimental to the lower ones. Freezing the lower layers can prevent performance degradation. Particularly, AlignFreeze improves Part-of-Speech (PoS) tagging performances in languages where full realignment fails: with XLM-R, it provides improvements of more than one standard deviation in accuracy in seven more languages than full realignment. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_12959 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | AlignFreeze: Navigating the Impact of Realignment on the Layers of Multilingual Models Across Diverse Languages Bakos, Steve Gaschi, Félix Guzmán, David More, Riddhi Li, Kelly Chutong Lee, En-Shiun Annie Computation and Language Artificial Intelligence Realignment techniques are often employed to enhance cross-lingual transfer in multilingual language models, still, they can sometimes degrade performance in languages that differ significantly from the fine-tuned source language. This paper introduces AlignFreeze, a method that freezes either the layers' lower half or upper half during realignment. Through controlled experiments on 4 tasks, 3 models, and in 35 languages, we find that realignment affects all the layers but can be the most detrimental to the lower ones. Freezing the lower layers can prevent performance degradation. Particularly, AlignFreeze improves Part-of-Speech (PoS) tagging performances in languages where full realignment fails: with XLM-R, it provides improvements of more than one standard deviation in accuracy in seven more languages than full realignment. |
| title | AlignFreeze: Navigating the Impact of Realignment on the Layers of Multilingual Models Across Diverse Languages |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2502.12959 |