Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2602.15210 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908852926021632 |
|---|---|
| author | DatologyAI : Carranza, Aldo Gael Mentzer, Kaleigh Monti, Ricardo Pio Fang, Alex Deng, Alvin Abbas, Amro Suri, Anshuman Larsen, Brett Blakeney, Cody Teh, Darren Schwab, David Kiner, Diego Pan, Fan Mongstad, Haakon Yin, Haoli Urbanek, Jack Lee, Jason Telanoff, Jason Wills, Josh Merrick, Luke Böther, Maximilian Doshi, Parth Burstein, Paul Maini, Pratyush Adiga, Rishabh Joshi, Siddharth Das, Spandan Jiang, Tony Dorna, Vineeth Wang, Zhengping Gaza, Bogdan Morcos, Ari Leavitt, Matthew |
| author_facet | DatologyAI : Carranza, Aldo Gael Mentzer, Kaleigh Monti, Ricardo Pio Fang, Alex Deng, Alvin Abbas, Amro Suri, Anshuman Larsen, Brett Blakeney, Cody Teh, Darren Schwab, David Kiner, Diego Pan, Fan Mongstad, Haakon Yin, Haoli Urbanek, Jack Lee, Jason Telanoff, Jason Wills, Josh Merrick, Luke Böther, Maximilian Doshi, Parth Burstein, Paul Maini, Pratyush Adiga, Rishabh Joshi, Siddharth Das, Spandan Jiang, Tony Dorna, Vineeth Wang, Zhengping Gaza, Bogdan Morcos, Ari Leavitt, Matthew |
| contents | Multilinguality is a core capability for modern foundation models, yet training high-quality multilingual models remains challenging due to uneven data availability across languages. A further challenge is the performance interference that can arise from joint multilingual training, commonly referred to as the "curse of multilinguality". We study multilingual data curation across thirteen languages and find that many reported regressions are not inherent to multilingual scaling but instead stem from correctable deficiencies in data quality and composition rather than fundamental capacity limits. In controlled bilingual experiments, improving data quality for any single language benefits others: curating English improves non-English performance in 12 of 13 languages, while curating non-English yields reciprocal improvements in English. Bespoke per-language curation produces substantially larger within-language improvements. Extending these findings to large-scale general-purpose training mixtures, we show that curated multilingual allocations comprising under 8% of total tokens remain remarkably effective. We operationalize this approach within an effort that produced a 20T-token pretraining corpus derived entirely from public sources. Models with 3B and 8B parameters trained on a 1T-token random subset achieve competitive multilingual accuracy with 4-10x fewer training FLOPs than strong public baselines, establishing a new Pareto frontier in multilingual performance versus compute. Moreover, these benefits extend to frontier model scale: the 20T-token corpus served as part of the pretraining dataset for Trinity Large (400B/A13B), which exhibits strong multilingual performance relative to its training FLOPs. These results show that targeted, per-language data curation mitigates multilingual interference and enables compute-efficient multilingual scaling. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_15210 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | ÜberWeb: Insights from Multilingual Curation for a 20-Trillion-Token Dataset DatologyAI : Carranza, Aldo Gael Mentzer, Kaleigh Monti, Ricardo Pio Fang, Alex Deng, Alvin Abbas, Amro Suri, Anshuman Larsen, Brett Blakeney, Cody Teh, Darren Schwab, David Kiner, Diego Pan, Fan Mongstad, Haakon Yin, Haoli Urbanek, Jack Lee, Jason Telanoff, Jason Wills, Josh Merrick, Luke Böther, Maximilian Doshi, Parth Burstein, Paul Maini, Pratyush Adiga, Rishabh Joshi, Siddharth Das, Spandan Jiang, Tony Dorna, Vineeth Wang, Zhengping Gaza, Bogdan Morcos, Ari Leavitt, Matthew Machine Learning Multilinguality is a core capability for modern foundation models, yet training high-quality multilingual models remains challenging due to uneven data availability across languages. A further challenge is the performance interference that can arise from joint multilingual training, commonly referred to as the "curse of multilinguality". We study multilingual data curation across thirteen languages and find that many reported regressions are not inherent to multilingual scaling but instead stem from correctable deficiencies in data quality and composition rather than fundamental capacity limits. In controlled bilingual experiments, improving data quality for any single language benefits others: curating English improves non-English performance in 12 of 13 languages, while curating non-English yields reciprocal improvements in English. Bespoke per-language curation produces substantially larger within-language improvements. Extending these findings to large-scale general-purpose training mixtures, we show that curated multilingual allocations comprising under 8% of total tokens remain remarkably effective. We operationalize this approach within an effort that produced a 20T-token pretraining corpus derived entirely from public sources. Models with 3B and 8B parameters trained on a 1T-token random subset achieve competitive multilingual accuracy with 4-10x fewer training FLOPs than strong public baselines, establishing a new Pareto frontier in multilingual performance versus compute. Moreover, these benefits extend to frontier model scale: the 20T-token corpus served as part of the pretraining dataset for Trinity Large (400B/A13B), which exhibits strong multilingual performance relative to its training FLOPs. These results show that targeted, per-language data curation mitigates multilingual interference and enables compute-efficient multilingual scaling. |
| title | ÜberWeb: Insights from Multilingual Curation for a 20-Trillion-Token Dataset |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2602.15210 |