Saved in:
| Main Authors: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2412.02595 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915314687541248 |
|---|---|
| author | Su, Dan Kong, Kezhi Lin, Ying Jennings, Joseph Norick, Brandon Kliegl, Markus Patwary, Mostofa Shoeybi, Mohammad Catanzaro, Bryan |
| author_facet | Su, Dan Kong, Kezhi Lin, Ying Jennings, Joseph Norick, Brandon Kliegl, Markus Patwary, Mostofa Shoeybi, Mohammad Catanzaro, Bryan |
| contents | Recent English Common Crawl datasets like FineWeb-Edu and DCLM achieved significant benchmark gains via aggressive model-based filtering, but at the cost of removing 90% of data. This limits their suitability for long token horizon training, such as 15T tokens for Llama 3.1. In this paper, we show how to achieve better trade-offs between accuracy and data quantity by a combination of classifier ensembling, synthetic data rephrasing, and reduced reliance on heuristic filters. When training 8B parameter models for 1T tokens, using a high-quality subset of our data improves MMLU by 5.6 over DCLM, demonstrating the efficacy of our methods for boosting accuracies over a relatively short token horizon. Furthermore, our full 6.3T token dataset matches DCLM on MMLU, but contains four times more unique real tokens than DCLM. This unlocks state-of-the-art training over a long token horizon: an 8B parameter model trained for 15T tokens, of which 7.2T came from our dataset, is better than the Llama 3.1 8B model: +5 on MMLU, +3.1 on ARC-Challenge, and +0.5 on average across ten diverse tasks. The dataset is available at https://data.commoncrawl.org/contrib/Nemotron/Nemotron-CC/index.html |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2412_02595 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Nemotron-CC: Transforming Common Crawl into a Refined Long-Horizon Pretraining Dataset Su, Dan Kong, Kezhi Lin, Ying Jennings, Joseph Norick, Brandon Kliegl, Markus Patwary, Mostofa Shoeybi, Mohammad Catanzaro, Bryan Computation and Language Recent English Common Crawl datasets like FineWeb-Edu and DCLM achieved significant benchmark gains via aggressive model-based filtering, but at the cost of removing 90% of data. This limits their suitability for long token horizon training, such as 15T tokens for Llama 3.1. In this paper, we show how to achieve better trade-offs between accuracy and data quantity by a combination of classifier ensembling, synthetic data rephrasing, and reduced reliance on heuristic filters. When training 8B parameter models for 1T tokens, using a high-quality subset of our data improves MMLU by 5.6 over DCLM, demonstrating the efficacy of our methods for boosting accuracies over a relatively short token horizon. Furthermore, our full 6.3T token dataset matches DCLM on MMLU, but contains four times more unique real tokens than DCLM. This unlocks state-of-the-art training over a long token horizon: an 8B parameter model trained for 15T tokens, of which 7.2T came from our dataset, is better than the Llama 3.1 8B model: +5 on MMLU, +3.1 on ARC-Challenge, and +0.5 on average across ten diverse tasks. The dataset is available at https://data.commoncrawl.org/contrib/Nemotron/Nemotron-CC/index.html |
| title | Nemotron-CC: Transforming Common Crawl into a Refined Long-Horizon Pretraining Dataset |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2412.02595 |