Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2402.19282 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913268555055104 |
|---|---|
| author | Qiu, Jiantao Lv, Haijun Jin, Zhenjiang Wang, Rui Ning, Wenchang Yu, Jia Zhang, ChaoBin Li, Zhenxiang Chu, Pei Qu, Yuan Shi, Jin Lu, Lindong Peng, Runyu Zeng, Zhiyuan Tang, Huanze Lei, Zhikai Hong, Jiawei Chen, Keyu Fei, Zhaoye Xu, Ruiliang Li, Wei Tu, Zhongying Dahua, Lin Qiao, Yu Yan, Hang He, Conghui |
| author_facet | Qiu, Jiantao Lv, Haijun Jin, Zhenjiang Wang, Rui Ning, Wenchang Yu, Jia Zhang, ChaoBin Li, Zhenxiang Chu, Pei Qu, Yuan Shi, Jin Lu, Lindong Peng, Runyu Zeng, Zhiyuan Tang, Huanze Lei, Zhikai Hong, Jiawei Chen, Keyu Fei, Zhaoye Xu, Ruiliang Li, Wei Tu, Zhongying Dahua, Lin Qiao, Yu Yan, Hang He, Conghui |
| contents | This paper presents WanJuan-CC, a safe and high-quality open-sourced English webtext dataset derived from Common Crawl data. The study addresses the challenges of constructing large-scale pre-training datasets for language models, which require vast amounts of high-quality data. A comprehensive process was designed to handle Common Crawl data, including extraction, heuristic rule filtering, fuzzy deduplication, content safety filtering, and data quality filtering. From approximately 68 billion original English documents, we obtained 2.22T Tokens of safe data and selected 1.0T Tokens of high-quality data as part of WanJuan-CC. We have open-sourced 100B Tokens from this dataset. The paper also provides statistical information related to data quality, enabling users to select appropriate data according to their needs. To evaluate the quality and utility of the dataset, we trained 1B-parameter and 3B-parameter models using WanJuan-CC and another dataset, RefinedWeb. Results show that WanJuan-CC performs better on validation datasets and downstream tasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_19282 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | WanJuan-CC: A Safe and High-Quality Open-sourced English Webtext Dataset Qiu, Jiantao Lv, Haijun Jin, Zhenjiang Wang, Rui Ning, Wenchang Yu, Jia Zhang, ChaoBin Li, Zhenxiang Chu, Pei Qu, Yuan Shi, Jin Lu, Lindong Peng, Runyu Zeng, Zhiyuan Tang, Huanze Lei, Zhikai Hong, Jiawei Chen, Keyu Fei, Zhaoye Xu, Ruiliang Li, Wei Tu, Zhongying Dahua, Lin Qiao, Yu Yan, Hang He, Conghui Computation and Language This paper presents WanJuan-CC, a safe and high-quality open-sourced English webtext dataset derived from Common Crawl data. The study addresses the challenges of constructing large-scale pre-training datasets for language models, which require vast amounts of high-quality data. A comprehensive process was designed to handle Common Crawl data, including extraction, heuristic rule filtering, fuzzy deduplication, content safety filtering, and data quality filtering. From approximately 68 billion original English documents, we obtained 2.22T Tokens of safe data and selected 1.0T Tokens of high-quality data as part of WanJuan-CC. We have open-sourced 100B Tokens from this dataset. The paper also provides statistical information related to data quality, enabling users to select appropriate data according to their needs. To evaluate the quality and utility of the dataset, we trained 1B-parameter and 3B-parameter models using WanJuan-CC and another dataset, RefinedWeb. Results show that WanJuan-CC performs better on validation datasets and downstream tasks. |
| title | WanJuan-CC: A Safe and High-Quality Open-sourced English Webtext Dataset |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2402.19282 |