Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | https://arxiv.org/abs/2501.14506 |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866913664608501760 |
|---|---|
| author | Yu, Jia Yuan, Fei Min, Rui Yu, Jing Chu, Pei Li, Jiayang Li, Wei Zhang, Ruijie Li, Zhenxiang Ren, Zhifei Zheng, Dong Zhang, Wenjian Teng, Yan Meng, Lingyu Jin, ZhenJiang Qiu, Jiantao Wang, ShaSha Tu, Zhongying Lin, Dahua Wang, Yu Qiao, Yu Wang, Yanfeng He, Conghui |
| author_facet | Yu, Jia Yuan, Fei Min, Rui Yu, Jing Chu, Pei Li, Jiayang Li, Wei Zhang, Ruijie Li, Zhenxiang Ren, Zhifei Zheng, Dong Zhang, Wenjian Teng, Yan Meng, Lingyu Jin, ZhenJiang Qiu, Jiantao Wang, ShaSha Tu, Zhongying Lin, Dahua Wang, Yu Qiao, Yu Wang, Yanfeng He, Conghui |
| contents | This paper introduces the open-source dataset WanJuanSiLu, designed to provide high-quality training corpora for low-resource languages, thereby advancing the research and development of multilingual models. To achieve this, we have developed a systematic data processing framework tailored for low-resource languages. This framework encompasses key stages such as data extraction, corpus cleaning, content deduplication, security filtering, quality evaluation, and theme classification. Through the implementation of this framework, we have significantly improved both the quality and security of the dataset, while maintaining its linguistic diversity. As of now, data for all five languages have been fully open-sourced. The dataset can be accessed at https://opendatalab.com/applyMultilingualCorpus, and GitHub repository is available at https://github.com/opendatalab/WanJuan3.0 |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_14506 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | WanJuanSiLu: A High-Quality Open-Source Webtext Dataset for Low-Resource Languages Yu, Jia Yuan, Fei Min, Rui Yu, Jing Chu, Pei Li, Jiayang Li, Wei Zhang, Ruijie Li, Zhenxiang Ren, Zhifei Zheng, Dong Zhang, Wenjian Teng, Yan Meng, Lingyu Jin, ZhenJiang Qiu, Jiantao Wang, ShaSha Tu, Zhongying Lin, Dahua Wang, Yu Qiao, Yu Wang, Yanfeng He, Conghui Computation and Language This paper introduces the open-source dataset WanJuanSiLu, designed to provide high-quality training corpora for low-resource languages, thereby advancing the research and development of multilingual models. To achieve this, we have developed a systematic data processing framework tailored for low-resource languages. This framework encompasses key stages such as data extraction, corpus cleaning, content deduplication, security filtering, quality evaluation, and theme classification. Through the implementation of this framework, we have significantly improved both the quality and security of the dataset, while maintaining its linguistic diversity. As of now, data for all five languages have been fully open-sourced. The dataset can be accessed at https://opendatalab.com/applyMultilingualCorpus, and GitHub repository is available at https://github.com/opendatalab/WanJuan3.0 |
| title | WanJuanSiLu: A High-Quality Open-Source Webtext Dataset for Low-Resource Languages |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2501.14506 |