Copy First, Translate Later: Interpreting Translation Dynamics in Multilingual Pretraining
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2026
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866915944740159488 |
|---|---|
| author | Körner, Felicia Matveev, Maria Eichin, Florian Kutyniok, Gitta Plank, Barbara Hedderich, Michael A. |
| author_facet | Körner, Felicia Matveev, Maria Eichin, Florian Kutyniok, Gitta Plank, Barbara Hedderich, Michael A. |
| contents | Large language models exhibit impressive cross-lingual capabilities. However, prior work analyzes this phenomenon through isolated factors and at sparse points during training, limiting our understanding of how cross-lingual generalization emerges--particularly in the early phases of learning. To study the early trajectory of linguistic and translation capabilities, we pretrain a multilingual 1.7B model on nine diverse languages, capturing checkpoints at a much finer granularity. We further introduce a novel word-level translation dataset and trace how translation develops over training through behavioral analyses, model-component analysis, and parameter-based ablations. We find that the model quickly acquires basic linguistic capabilities in parallel with token-level copying, while translation develops in two distinct phases: an initial phase dominated by copying and surface-level similarities, and a second phase in which more generalizing translation mechanisms are developed while copying is refined. Together, these findings provide a fine-grained view of how cross-lingual generalization develops during multilingual pretraining. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_17633 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Copy First, Translate Later: Interpreting Translation Dynamics in Multilingual Pretraining Körner, Felicia Matveev, Maria Eichin, Florian Kutyniok, Gitta Plank, Barbara Hedderich, Michael A. Computation and Language Large language models exhibit impressive cross-lingual capabilities. However, prior work analyzes this phenomenon through isolated factors and at sparse points during training, limiting our understanding of how cross-lingual generalization emerges--particularly in the early phases of learning. To study the early trajectory of linguistic and translation capabilities, we pretrain a multilingual 1.7B model on nine diverse languages, capturing checkpoints at a much finer granularity. We further introduce a novel word-level translation dataset and trace how translation develops over training through behavioral analyses, model-component analysis, and parameter-based ablations. We find that the model quickly acquires basic linguistic capabilities in parallel with token-level copying, while translation develops in two distinct phases: an initial phase dominated by copying and surface-level similarities, and a second phase in which more generalizing translation mechanisms are developed while copying is refined. Together, these findings provide a fine-grained view of how cross-lingual generalization develops during multilingual pretraining. |
| title | Copy First, Translate Later: Interpreting Translation Dynamics in Multilingual Pretraining |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2604.17633 |