Do Language Models Care About Text Quality? Evaluating Web-Crawled Corpora Across 11 Languages
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866929275099152384 |
|---|---|
| author | van Noord, Rik Kuzman, Taja Rupnik, Peter Ljubešić, Nikola Esplà-Gomis, Miquel Ramírez-Sánchez, Gema Toral, Antonio |
| author_facet | van Noord, Rik Kuzman, Taja Rupnik, Peter Ljubešić, Nikola Esplà-Gomis, Miquel Ramírez-Sánchez, Gema Toral, Antonio |
| contents | Large, curated, web-crawled corpora play a vital role in training language models (LMs). They form the lion's share of the training data in virtually all recent LMs, such as the well-known GPT, LLaMA and XLM-RoBERTa models. However, despite this importance, relatively little attention has been given to the quality of these corpora. In this paper, we compare four of the currently most relevant large, web-crawled corpora (CC100, MaCoCu, mC4 and OSCAR) across eleven lower-resourced European languages. Our approach is two-fold: first, we perform an intrinsic evaluation by performing a human evaluation of the quality of samples taken from different corpora; then, we assess the practical impact of the qualitative differences by training specific LMs on each of the corpora and evaluating their performance on downstream tasks. We find that there are clear differences in quality of the corpora, with MaCoCu and OSCAR obtaining the best results. However, during the extrinsic evaluation, we actually find that the CC100 corpus achieves the highest scores. We conclude that, in our experiments, the quality of the web-crawled corpora does not seem to play a significant role when training LMs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_08693 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Do Language Models Care About Text Quality? Evaluating Web-Crawled Corpora Across 11 Languages van Noord, Rik Kuzman, Taja Rupnik, Peter Ljubešić, Nikola Esplà-Gomis, Miquel Ramírez-Sánchez, Gema Toral, Antonio Computation and Language Large, curated, web-crawled corpora play a vital role in training language models (LMs). They form the lion's share of the training data in virtually all recent LMs, such as the well-known GPT, LLaMA and XLM-RoBERTa models. However, despite this importance, relatively little attention has been given to the quality of these corpora. In this paper, we compare four of the currently most relevant large, web-crawled corpora (CC100, MaCoCu, mC4 and OSCAR) across eleven lower-resourced European languages. Our approach is two-fold: first, we perform an intrinsic evaluation by performing a human evaluation of the quality of samples taken from different corpora; then, we assess the practical impact of the qualitative differences by training specific LMs on each of the corpora and evaluating their performance on downstream tasks. We find that there are clear differences in quality of the corpora, with MaCoCu and OSCAR obtaining the best results. However, during the extrinsic evaluation, we actually find that the CC100 corpus achieves the highest scores. We conclude that, in our experiments, the quality of the web-crawled corpora does not seem to play a significant role when training LMs. |
| title | Do Language Models Care About Text Quality? Evaluating Web-Crawled Corpora Across 11 Languages |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2403.08693 |