Beyond the Rosetta Stone: Unification Forces in Generalization Dynamics
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866915467509104640 |
|---|---|
| author | Blum, Carter Filippova, Katja Yuan, Ann Ghandeharioun, Asma Zimmert, Julian Zhang, Fred Hoffmann, Jessica Linzen, Tal Wattenberg, Martin Dixon, Lucas Geva, Mor |
| author_facet | Blum, Carter Filippova, Katja Yuan, Ann Ghandeharioun, Asma Zimmert, Julian Zhang, Fred Hoffmann, Jessica Linzen, Tal Wattenberg, Martin Dixon, Lucas Geva, Mor |
| contents | Large language models (LLMs) struggle with cross-lingual knowledge transfer: they hallucinate when asked in one language about facts expressed in a different language during training. This work introduces a controlled setting to study the causes and dynamics of this phenomenon by training small Transformer models from scratch on synthetic multilingual datasets. We identify a learning phase wherein a model develops either separate or unified representations of the same facts across languages, and show that unification is essential for cross-lingual transfer. We also show that the degree of unification depends on mutual information between facts and training data language, and on how easy it is to extract that language. Based on these insights, we develop methods to modulate the level of cross-lingual transfer by manipulating data distribution and tokenization, and we introduce metrics and visualizations to formally characterize their effects on unification. Our work shows how controlled settings can shed light on pre-training dynamics and suggests new directions for improving cross-lingual transfer in LLMs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_11017 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Beyond the Rosetta Stone: Unification Forces in Generalization Dynamics Blum, Carter Filippova, Katja Yuan, Ann Ghandeharioun, Asma Zimmert, Julian Zhang, Fred Hoffmann, Jessica Linzen, Tal Wattenberg, Martin Dixon, Lucas Geva, Mor Computation and Language Artificial Intelligence Large language models (LLMs) struggle with cross-lingual knowledge transfer: they hallucinate when asked in one language about facts expressed in a different language during training. This work introduces a controlled setting to study the causes and dynamics of this phenomenon by training small Transformer models from scratch on synthetic multilingual datasets. We identify a learning phase wherein a model develops either separate or unified representations of the same facts across languages, and show that unification is essential for cross-lingual transfer. We also show that the degree of unification depends on mutual information between facts and training data language, and on how easy it is to extract that language. Based on these insights, we develop methods to modulate the level of cross-lingual transfer by manipulating data distribution and tokenization, and we introduce metrics and visualizations to formally characterize their effects on unification. Our work shows how controlled settings can shed light on pre-training dynamics and suggests new directions for improving cross-lingual transfer in LLMs. |
| title | Beyond the Rosetta Stone: Unification Forces in Generalization Dynamics |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2508.11017 |