FuLG: 150B Romanian Corpus for Language Model Pretraining
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866929426617335808 |
|---|---|
| author | Bădoiu, Vlad-Andrei Dumitru, Mihai-Valentin Gherghescu, Alexandru M. Agache, Alexandru Raiciu, Costin |
| author_facet | Bădoiu, Vlad-Andrei Dumitru, Mihai-Valentin Gherghescu, Alexandru M. Agache, Alexandru Raiciu, Costin |
| contents | Research in the field of language models is rapidly evolving, with many open models being released to the public. Openly available pretraining corpora usually focus on only a handful of languages, with many others either missing completely or extremely underrepresented. In this report, we introduce FuLG, a hundred-fifty-billion-token Romanian corpus extracted from CommonCrawl. We present our methodology for filtering FuLG and compare it via ablation studies against existing Romanian corpora. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_13657 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | FuLG: 150B Romanian Corpus for Language Model Pretraining Bădoiu, Vlad-Andrei Dumitru, Mihai-Valentin Gherghescu, Alexandru M. Agache, Alexandru Raiciu, Costin Computation and Language Research in the field of language models is rapidly evolving, with many open models being released to the public. Openly available pretraining corpora usually focus on only a handful of languages, with many others either missing completely or extremely underrepresented. In this report, we introduce FuLG, a hundred-fifty-billion-token Romanian corpus extracted from CommonCrawl. We present our methodology for filtering FuLG and compare it via ablation studies against existing Romanian corpora. |
| title | FuLG: 150B Romanian Corpus for Language Model Pretraining |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2407.13657 |