GlotCC: An Open Broad-Coverage CommonCrawl Corpus and Pipeline for Minority Languages
Fuente:
arXiv
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909523436896256 |
|---|---|
| author | Kargaran, Amir Hossein Yvon, François Schütze, Hinrich |
| author_facet | Kargaran, Amir Hossein Yvon, François Schütze, Hinrich |
| contents | The need for large text corpora has increased with the advent of pretrained language models and, in particular, the discovery of scaling laws for these models. Most available corpora have sufficient data only for languages with large dominant communities. However, there is no corpus available that (i) covers a wide range of minority languages; (ii) is generated by an open-source reproducible pipeline; and (iii) is rigorously cleaned from noise, making it trustworthy to use. We present GlotCC, a clean, document-level, 2TB general domain corpus derived from CommonCrawl, covering more than 1000 languages. We make GlotCC and the system used to generate it - including the pipeline, language identification model, and filters - available to the research community. Corpus v. 1.0 https://huggingface.co/datasets/cis-lmu/GlotCC-v1, Pipeline v. 3.0 https://github.com/cisnlp/GlotCC. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_23825 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | GlotCC: An Open Broad-Coverage CommonCrawl Corpus and Pipeline for Minority Languages Kargaran, Amir Hossein Yvon, François Schütze, Hinrich Computation and Language Artificial Intelligence The need for large text corpora has increased with the advent of pretrained language models and, in particular, the discovery of scaling laws for these models. Most available corpora have sufficient data only for languages with large dominant communities. However, there is no corpus available that (i) covers a wide range of minority languages; (ii) is generated by an open-source reproducible pipeline; and (iii) is rigorously cleaned from noise, making it trustworthy to use. We present GlotCC, a clean, document-level, 2TB general domain corpus derived from CommonCrawl, covering more than 1000 languages. We make GlotCC and the system used to generate it - including the pipeline, language identification model, and filters - available to the research community. Corpus v. 1.0 https://huggingface.co/datasets/cis-lmu/GlotCC-v1, Pipeline v. 3.0 https://github.com/cisnlp/GlotCC. |
| title | GlotCC: An Open Broad-Coverage CommonCrawl Corpus and Pipeline for Minority Languages |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2410.23825 |