Data Processing for the OpenGPT-X Model Family
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866913978105462784 |
|---|---|
| author | Brandizzi, Nicolo' Abdelwahab, Hammam Bhowmick, Anirban Helmer, Lennard Stein, Benny Jörg Denisov, Pavel Saleem, Qasid Fromm, Michael Ali, Mehdi Rutmann, Richard Naderi, Farzad Agy, Mohamad Saif Schwirjow, Alexander Küch, Fabian Hahn, Luzian Ostendorff, Malte Suarez, Pedro Ortiz Rehm, Georg Wegener, Dennis Flores-Herr, Nicolas Köhler, Joachim Leveling, Johannes |
| author_facet | Brandizzi, Nicolo' Abdelwahab, Hammam Bhowmick, Anirban Helmer, Lennard Stein, Benny Jörg Denisov, Pavel Saleem, Qasid Fromm, Michael Ali, Mehdi Rutmann, Richard Naderi, Farzad Agy, Mohamad Saif Schwirjow, Alexander Küch, Fabian Hahn, Luzian Ostendorff, Malte Suarez, Pedro Ortiz Rehm, Georg Wegener, Dennis Flores-Herr, Nicolas Köhler, Joachim Leveling, Johannes |
| contents | This paper presents a comprehensive overview of the data preparation pipeline developed for the OpenGPT-X project, a large-scale initiative aimed at creating open and high-performance multilingual large language models (LLMs). The project goal is to deliver models that cover all major European languages, with a particular focus on real-world applications within the European Union. We explain all data processing steps, starting with the data selection and requirement definition to the preparation of the final filtered data. We distinguish between curated data and web data, as each of these categories is handled by distinct pipelines, with curated data undergoing minimal filtering and web data requiring extensive filtering and deduplication. This distinction guided the development of specialized algorithmic solutions for both pipelines. In addition to describing the processing methodologies, we provide an in-depth analysis of the datasets, increasing transparency and alignment with European data regulations. Finally, we share key insights and challenges faced during the project, offering recommendations for future endeavors in large-scale multilingual data preparation for LLMs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_08800 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Data Processing for the OpenGPT-X Model Family Brandizzi, Nicolo' Abdelwahab, Hammam Bhowmick, Anirban Helmer, Lennard Stein, Benny Jörg Denisov, Pavel Saleem, Qasid Fromm, Michael Ali, Mehdi Rutmann, Richard Naderi, Farzad Agy, Mohamad Saif Schwirjow, Alexander Küch, Fabian Hahn, Luzian Ostendorff, Malte Suarez, Pedro Ortiz Rehm, Georg Wegener, Dennis Flores-Herr, Nicolas Köhler, Joachim Leveling, Johannes Computation and Language H.3.1; I.2.7 This paper presents a comprehensive overview of the data preparation pipeline developed for the OpenGPT-X project, a large-scale initiative aimed at creating open and high-performance multilingual large language models (LLMs). The project goal is to deliver models that cover all major European languages, with a particular focus on real-world applications within the European Union. We explain all data processing steps, starting with the data selection and requirement definition to the preparation of the final filtered data. We distinguish between curated data and web data, as each of these categories is handled by distinct pipelines, with curated data undergoing minimal filtering and web data requiring extensive filtering and deduplication. This distinction guided the development of specialized algorithmic solutions for both pipelines. In addition to describing the processing methodologies, we provide an in-depth analysis of the datasets, increasing transparency and alignment with European data regulations. Finally, we share key insights and challenges faced during the project, offering recommendations for future endeavors in large-scale multilingual data preparation for LLMs. |
| title | Data Processing for the OpenGPT-X Model Family |
| topic | Computation and Language H.3.1; I.2.7 |
| url | https://arxiv.org/abs/2410.08800 |