Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2509.15556 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908546739732480 |
|---|---|
| author | Guo, Ping Ren, Yubing Liu, Binbin Liu, Fengze Lin, Haobin Zhang, Yifan Zhang, Bingni Wang, Taifeng Zheng, Yin |
| author_facet | Guo, Ping Ren, Yubing Liu, Binbin Liu, Fengze Lin, Haobin Zhang, Yifan Zhang, Bingni Wang, Taifeng Zheng, Yin |
| contents | Large language models (LLMs) have become integral to a wide range of applications worldwide, driving an unprecedented global demand for effective multilingual capabilities. Central to achieving robust multilingual performance is the strategic allocation of language proportions within training corpora. However, determining optimal language ratios is highly challenging due to intricate cross-lingual interactions and sensitivity to dataset scale. This paper introduces Climb (Cross-Lingual Interaction-aware Multilingual Balancing), a novel framework designed to systematically optimize multilingual data allocation. At its core, Climb introduces a cross-lingual interaction-aware language ratio, explicitly quantifying each language's effective allocation by capturing inter-language dependencies. Leveraging this ratio, Climb proposes a principled two-step optimization procedure--first equalizing marginal benefits across languages, then maximizing the magnitude of the resulting language allocation vectors--significantly simplifying the inherently complex multilingual optimization problem. Extensive experiments confirm that Climb can accurately measure cross-lingual interactions across various multilingual settings. LLMs trained with Climb-derived proportions consistently achieve state-of-the-art multilingual performance, even achieving competitive performance with open-sourced LLMs trained with more tokens. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_15556 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Exploring Polyglot Harmony: On Multilingual Data Allocation for Large Language Models Pretraining Guo, Ping Ren, Yubing Liu, Binbin Liu, Fengze Lin, Haobin Zhang, Yifan Zhang, Bingni Wang, Taifeng Zheng, Yin Computation and Language Artificial Intelligence Large language models (LLMs) have become integral to a wide range of applications worldwide, driving an unprecedented global demand for effective multilingual capabilities. Central to achieving robust multilingual performance is the strategic allocation of language proportions within training corpora. However, determining optimal language ratios is highly challenging due to intricate cross-lingual interactions and sensitivity to dataset scale. This paper introduces Climb (Cross-Lingual Interaction-aware Multilingual Balancing), a novel framework designed to systematically optimize multilingual data allocation. At its core, Climb introduces a cross-lingual interaction-aware language ratio, explicitly quantifying each language's effective allocation by capturing inter-language dependencies. Leveraging this ratio, Climb proposes a principled two-step optimization procedure--first equalizing marginal benefits across languages, then maximizing the magnitude of the resulting language allocation vectors--significantly simplifying the inherently complex multilingual optimization problem. Extensive experiments confirm that Climb can accurately measure cross-lingual interactions across various multilingual settings. LLMs trained with Climb-derived proportions consistently achieve state-of-the-art multilingual performance, even achieving competitive performance with open-sourced LLMs trained with more tokens. |
| title | Exploring Polyglot Harmony: On Multilingual Data Allocation for Large Language Models Pretraining |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2509.15556 |