Zyda-2: a 5 Trillion Token High-Quality Dataset
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866910692409344000 |
|---|---|
| author | Tokpanov, Yury Glorioso, Paolo Anthony, Quentin Millidge, Beren |
| author_facet | Tokpanov, Yury Glorioso, Paolo Anthony, Quentin Millidge, Beren |
| contents | In this technical report, we present Zyda-2: a five trillion token dataset for language model pretraining. Zyda-2 was used to train our Zamba2 series of models which are state-of-the-art for their weight class. We build Zyda-2 by collating high-quality open-source tokens such as FineWeb and DCLM, then distilling them to the highest-quality subset via cross-deduplication and model-based quality filtering. Zyda-2 is released under a permissive open license, and is available at https://huggingface.co/datasets/Zyphra/Zyda-2 |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_06068 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Zyda-2: a 5 Trillion Token High-Quality Dataset Tokpanov, Yury Glorioso, Paolo Anthony, Quentin Millidge, Beren Computation and Language Artificial Intelligence In this technical report, we present Zyda-2: a five trillion token dataset for language model pretraining. Zyda-2 was used to train our Zamba2 series of models which are state-of-the-art for their weight class. We build Zyda-2 by collating high-quality open-source tokens such as FineWeb and DCLM, then distilling them to the highest-quality subset via cross-deduplication and model-based quality filtering. Zyda-2 is released under a permissive open license, and is available at https://huggingface.co/datasets/Zyphra/Zyda-2 |
| title | Zyda-2: a 5 Trillion Token High-Quality Dataset |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2411.06068 |