Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2510.13996 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909849602752512 |
|---|---|
| author | Gienapp, Lukas Schröder, Christopher Schweter, Stefan Akiki, Christopher Schlatt, Ferdinand Zimmermann, Arden Genêt, Phillipe Potthast, Martin |
| author_facet | Gienapp, Lukas Schröder, Christopher Schweter, Stefan Akiki, Christopher Schlatt, Ferdinand Zimmermann, Arden Genêt, Phillipe Potthast, Martin |
| contents | Large language model development relies on large-scale training corpora, yet most contain data of unclear licensing status, limiting the development of truly open models. This problem is exacerbated for non-English languages, where openly licensed text remains critically scarce. We introduce the German Commons, the largest collection of openly licensed German text to date. It compiles data from 41 sources across seven domains, encompassing legal, scientific, cultural, political, news, economic, and web text. Through systematic sourcing from established data providers with verifiable licensing, it yields 154.56 billion tokens of high-quality text for language model training. Our processing pipeline implements comprehensive quality filtering, deduplication, and text formatting fixes, ensuring consistent quality across heterogeneous text sources. All domain subsets feature licenses of at least CC-BY-SA 4.0 or equivalent, ensuring legal compliance for model training and redistribution. The German Commons therefore addresses the critical gap in openly licensed German pretraining data, and enables the development of truly open German language models. We also release code for corpus construction and data filtering tailored to German language text, rendering the German Commons fully reproducible and extensible. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_13996 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | The German Commons - 154 Billion Tokens of Openly Licensed Text for German Language Models Gienapp, Lukas Schröder, Christopher Schweter, Stefan Akiki, Christopher Schlatt, Ferdinand Zimmermann, Arden Genêt, Phillipe Potthast, Martin Computation and Language Large language model development relies on large-scale training corpora, yet most contain data of unclear licensing status, limiting the development of truly open models. This problem is exacerbated for non-English languages, where openly licensed text remains critically scarce. We introduce the German Commons, the largest collection of openly licensed German text to date. It compiles data from 41 sources across seven domains, encompassing legal, scientific, cultural, political, news, economic, and web text. Through systematic sourcing from established data providers with verifiable licensing, it yields 154.56 billion tokens of high-quality text for language model training. Our processing pipeline implements comprehensive quality filtering, deduplication, and text formatting fixes, ensuring consistent quality across heterogeneous text sources. All domain subsets feature licenses of at least CC-BY-SA 4.0 or equivalent, ensuring legal compliance for model training and redistribution. The German Commons therefore addresses the critical gap in openly licensed German pretraining data, and enables the development of truly open German language models. We also release code for corpus construction and data filtering tailored to German language text, rendering the German Commons fully reproducible and extensible. |
| title | The German Commons - 154 Billion Tokens of Openly Licensed Text for German Language Models |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2510.13996 |