HPLT 3.0: Very Large-Scale Multilingual Resources for LLMs and MT. Mono- and Bi-lingual Data, Multilingual Evaluation, and Pre-Trained Models
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915944052293632 |
|---|---|
| author | Oepen, Stephan Arefev, Nikolay Aulamo, Mikko Bañón, Marta Buljan, Maja Burchell, Laurie Charpentier, Lucas Chen, Pinzhen Fedorova, Mariya de Gibert, Ona Haddow, Barry Hajič, Jan Helcl, Jindřich Kutuzov, Andrey Laippala, Veronika Li, Zihao Luukkonen, Risto Malik, Bhavitvya Mikhailov, Vladislav Myntti, Amanda O'Brien, Dayyán Poláková, Lucie Pyysalo, Sampo Sánchez, Gema Ramírez Siewert, Janine Stepachev, Pavel Tiedemann, Jörg Vahtola, Teemu Variš, Dušan Vitiugin, Fedor Vojtěchová, Tea Zaragoza, Jaume |
| author_facet | Oepen, Stephan Arefev, Nikolay Aulamo, Mikko Bañón, Marta Buljan, Maja Burchell, Laurie Charpentier, Lucas Chen, Pinzhen Fedorova, Mariya de Gibert, Ona Haddow, Barry Hajič, Jan Helcl, Jindřich Kutuzov, Andrey Laippala, Veronika Li, Zihao Luukkonen, Risto Malik, Bhavitvya Mikhailov, Vladislav Myntti, Amanda O'Brien, Dayyán Poláková, Lucie Pyysalo, Sampo Sánchez, Gema Ramírez Siewert, Janine Stepachev, Pavel Tiedemann, Jörg Vahtola, Teemu Variš, Dušan Vitiugin, Fedor Vojtěchová, Tea Zaragoza, Jaume |
| contents | We present an ongoing initiative to provide open, very large, high-quality, and richly annotated textual datasets for almost 200 languages. At 30 trillion tokens, this is likely the largest generally available multilingual collection of LLM pre-training data. These datasets are derived from web crawls from different sources and accompanied with a complete, open-source pipeline for document selection from web archives, text extraction from HTML, language identification for noisy texts, exact and near-deduplication, annotation with, among others, register labels, text quality estimates, and personally identifiable information; and final selection and filtering. We report on data quality probes through contrastive and analytical statistics, through manual inspection of samples for 24 languages, and through end-to-end evaluation of various language model architectures trained on this data. For multilingual LLM evaluation, we provide a comprehensive collection of benchmarks for nine European languages, with special emphasis on natively created tasks, mechanisms to mitigate prompt sensitivity, and refined normalization and aggregation of scores. Additionally, we train and evaluate a family of 57 monolingual encoder-decoder models, as well as a handful of monolingual GPT-like reference models. Besides the monolingual data and models, we also present a very large collection of parallel texts automatically mined from this data, together with a novel parallel corpus synthesized via machine translation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_01066 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | HPLT 3.0: Very Large-Scale Multilingual Resources for LLMs and MT. Mono- and Bi-lingual Data, Multilingual Evaluation, and Pre-Trained Models Oepen, Stephan Arefev, Nikolay Aulamo, Mikko Bañón, Marta Buljan, Maja Burchell, Laurie Charpentier, Lucas Chen, Pinzhen Fedorova, Mariya de Gibert, Ona Haddow, Barry Hajič, Jan Helcl, Jindřich Kutuzov, Andrey Laippala, Veronika Li, Zihao Luukkonen, Risto Malik, Bhavitvya Mikhailov, Vladislav Myntti, Amanda O'Brien, Dayyán Poláková, Lucie Pyysalo, Sampo Sánchez, Gema Ramírez Siewert, Janine Stepachev, Pavel Tiedemann, Jörg Vahtola, Teemu Variš, Dušan Vitiugin, Fedor Vojtěchová, Tea Zaragoza, Jaume Computation and Language We present an ongoing initiative to provide open, very large, high-quality, and richly annotated textual datasets for almost 200 languages. At 30 trillion tokens, this is likely the largest generally available multilingual collection of LLM pre-training data. These datasets are derived from web crawls from different sources and accompanied with a complete, open-source pipeline for document selection from web archives, text extraction from HTML, language identification for noisy texts, exact and near-deduplication, annotation with, among others, register labels, text quality estimates, and personally identifiable information; and final selection and filtering. We report on data quality probes through contrastive and analytical statistics, through manual inspection of samples for 24 languages, and through end-to-end evaluation of various language model architectures trained on this data. For multilingual LLM evaluation, we provide a comprehensive collection of benchmarks for nine European languages, with special emphasis on natively created tasks, mechanisms to mitigate prompt sensitivity, and refined normalization and aggregation of scores. Additionally, we train and evaluate a family of 57 monolingual encoder-decoder models, as well as a handful of monolingual GPT-like reference models. Besides the monolingual data and models, we also present a very large collection of parallel texts automatically mined from this data, together with a novel parallel corpus synthesized via machine translation. |
| title | HPLT 3.0: Very Large-Scale Multilingual Resources for LLMs and MT. Mono- and Bi-lingual Data, Multilingual Evaluation, and Pre-Trained Models |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2511.01066 |