An Expanded Massive Multilingual Dataset for High-Performance Language Technologies (HPLT)
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866910985380429824 |
|---|---|
| author | Burchell, Laurie de Gibert, Ona Arefyev, Nikolay Aulamo, Mikko Bañón, Marta Chen, Pinzhen Fedorova, Mariia Guillou, Liane Haddow, Barry Hajič, Jan Helcl, Jindřich Henriksson, Erik Klimaszewski, Mateusz Komulainen, Ville Kutuzov, Andrey Kytöniemi, Joona Laippala, Veronika Mæhlum, Petter Malik, Bhavitvya Mehryary, Farrokh Mikhailov, Vladislav Moghe, Nikita Myntti, Amanda O'Brien, Dayyán Oepen, Stephan Pal, Proyag Piha, Jousia Pyysalo, Sampo Ramírez-Sánchez, Gema Samuel, David Stepachev, Pavel Tiedemann, Jörg Variš, Dušan Vojtěchová, Tereza Zaragoza-Bernabeu, Jaume |
| author_facet | Burchell, Laurie de Gibert, Ona Arefyev, Nikolay Aulamo, Mikko Bañón, Marta Chen, Pinzhen Fedorova, Mariia Guillou, Liane Haddow, Barry Hajič, Jan Helcl, Jindřich Henriksson, Erik Klimaszewski, Mateusz Komulainen, Ville Kutuzov, Andrey Kytöniemi, Joona Laippala, Veronika Mæhlum, Petter Malik, Bhavitvya Mehryary, Farrokh Mikhailov, Vladislav Moghe, Nikita Myntti, Amanda O'Brien, Dayyán Oepen, Stephan Pal, Proyag Piha, Jousia Pyysalo, Sampo Ramírez-Sánchez, Gema Samuel, David Stepachev, Pavel Tiedemann, Jörg Variš, Dušan Vojtěchová, Tereza Zaragoza-Bernabeu, Jaume |
| contents | Training state-of-the-art large language models requires vast amounts of clean and diverse textual data. However, building suitable multilingual datasets remains a challenge. In this work, we present HPLT v2, a collection of high-quality multilingual monolingual and parallel corpora, extending prior work of the HPLT project. The monolingual portion of the data contains 8T tokens covering 193 languages, while the parallel data contains 380M sentence pairs covering 51 languages. We document the entire data pipeline and release the code to reproduce it. We provide extensive analysis of the quality and characteristics of our data. Finally, we evaluate the performance of language models and machine translation systems trained on HPLT v2, demonstrating its value. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_10267 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | An Expanded Massive Multilingual Dataset for High-Performance Language Technologies (HPLT) Burchell, Laurie de Gibert, Ona Arefyev, Nikolay Aulamo, Mikko Bañón, Marta Chen, Pinzhen Fedorova, Mariia Guillou, Liane Haddow, Barry Hajič, Jan Helcl, Jindřich Henriksson, Erik Klimaszewski, Mateusz Komulainen, Ville Kutuzov, Andrey Kytöniemi, Joona Laippala, Veronika Mæhlum, Petter Malik, Bhavitvya Mehryary, Farrokh Mikhailov, Vladislav Moghe, Nikita Myntti, Amanda O'Brien, Dayyán Oepen, Stephan Pal, Proyag Piha, Jousia Pyysalo, Sampo Ramírez-Sánchez, Gema Samuel, David Stepachev, Pavel Tiedemann, Jörg Variš, Dušan Vojtěchová, Tereza Zaragoza-Bernabeu, Jaume Computation and Language Training state-of-the-art large language models requires vast amounts of clean and diverse textual data. However, building suitable multilingual datasets remains a challenge. In this work, we present HPLT v2, a collection of high-quality multilingual monolingual and parallel corpora, extending prior work of the HPLT project. The monolingual portion of the data contains 8T tokens covering 193 languages, while the parallel data contains 380M sentence pairs covering 51 languages. We document the entire data pipeline and release the code to reproduce it. We provide extensive analysis of the quality and characteristics of our data. Finally, we evaluate the performance of language models and machine translation systems trained on HPLT v2, demonstrating its value. |
| title | An Expanded Massive Multilingual Dataset for High-Performance Language Technologies (HPLT) |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2503.10267 |