A Post-trainer's Guide to Multilingual Training Data: Uncovering Cross-lingual Transfer Dynamics
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909589990014976 |
|---|---|
| author | Shimabucoro, Luisa Ustun, Ahmet Fadaee, Marzieh Ruder, Sebastian |
| author_facet | Shimabucoro, Luisa Ustun, Ahmet Fadaee, Marzieh Ruder, Sebastian |
| contents | In order for large language models to be useful across the globe, they are fine-tuned to follow instructions on multilingual data. Despite the ubiquity of such post-training, a clear understanding of the dynamics that enable cross-lingual transfer remains elusive. This study examines cross-lingual transfer (CLT) dynamics in realistic post-training settings. We study two model families of up to 35B parameters in size trained on carefully controlled mixtures of multilingual data on three generative tasks with varying levels of complexity (summarization, instruction following, and mathematical reasoning) in both single-task and multi-task instruction tuning settings. Overall, we find that the dynamics of cross-lingual transfer and multilingual performance cannot be explained by isolated variables, varying depending on the combination of post-training settings. Finally, we identify the conditions that lead to effective cross-lingual transfer in practice. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_16677 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | A Post-trainer's Guide to Multilingual Training Data: Uncovering Cross-lingual Transfer Dynamics Shimabucoro, Luisa Ustun, Ahmet Fadaee, Marzieh Ruder, Sebastian Computation and Language Artificial Intelligence In order for large language models to be useful across the globe, they are fine-tuned to follow instructions on multilingual data. Despite the ubiquity of such post-training, a clear understanding of the dynamics that enable cross-lingual transfer remains elusive. This study examines cross-lingual transfer (CLT) dynamics in realistic post-training settings. We study two model families of up to 35B parameters in size trained on carefully controlled mixtures of multilingual data on three generative tasks with varying levels of complexity (summarization, instruction following, and mathematical reasoning) in both single-task and multi-task instruction tuning settings. Overall, we find that the dynamics of cross-lingual transfer and multilingual performance cannot be explained by isolated variables, varying depending on the combination of post-training settings. Finally, we identify the conditions that lead to effective cross-lingual transfer in practice. |
| title | A Post-trainer's Guide to Multilingual Training Data: Uncovering Cross-lingual Transfer Dynamics |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2504.16677 |