Data, Data Everywhere: A Guide for Pretraining Dataset Construction
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866929551062335488 |
|---|---|
| author | Parmar, Jupinder Prabhumoye, Shrimai Jennings, Joseph Liu, Bo Jhunjhunwala, Aastha Wang, Zhilin Patwary, Mostofa Shoeybi, Mohammad Catanzaro, Bryan |
| author_facet | Parmar, Jupinder Prabhumoye, Shrimai Jennings, Joseph Liu, Bo Jhunjhunwala, Aastha Wang, Zhilin Patwary, Mostofa Shoeybi, Mohammad Catanzaro, Bryan |
| contents | The impressive capabilities of recent language models can be largely attributed to the multi-trillion token pretraining datasets that they are trained on. However, model developers fail to disclose their construction methodology which has lead to a lack of open information on how to develop effective pretraining sets. To address this issue, we perform the first systematic study across the entire pipeline of pretraining set construction. First, we run ablations on existing techniques for pretraining set development to identify which methods translate to the largest gains in model accuracy on downstream evaluations. Then, we categorize the most widely used data source, web crawl snapshots, across the attributes of toxicity, quality, type of speech, and domain. Finally, we show how such attribute information can be used to further refine and improve the quality of a pretraining set. These findings constitute an actionable set of steps that practitioners can use to develop high quality pretraining sets. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_06380 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Data, Data Everywhere: A Guide for Pretraining Dataset Construction Parmar, Jupinder Prabhumoye, Shrimai Jennings, Joseph Liu, Bo Jhunjhunwala, Aastha Wang, Zhilin Patwary, Mostofa Shoeybi, Mohammad Catanzaro, Bryan Computation and Language The impressive capabilities of recent language models can be largely attributed to the multi-trillion token pretraining datasets that they are trained on. However, model developers fail to disclose their construction methodology which has lead to a lack of open information on how to develop effective pretraining sets. To address this issue, we perform the first systematic study across the entire pipeline of pretraining set construction. First, we run ablations on existing techniques for pretraining set development to identify which methods translate to the largest gains in model accuracy on downstream evaluations. Then, we categorize the most widely used data source, web crawl snapshots, across the attributes of toxicity, quality, type of speech, and domain. Finally, we show how such attribute information can be used to further refine and improve the quality of a pretraining set. These findings constitute an actionable set of steps that practitioners can use to develop high quality pretraining sets. |
| title | Data, Data Everywhere: A Guide for Pretraining Dataset Construction |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2407.06380 |