An Integrated Data Processing Framework for Pretraining Foundation Models
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866914765358497792 |
|---|---|
| author | Sun, Yiding Wang, Feng Zhu, Yutao Zhao, Wayne Xin Mao, Jiaxin |
| author_facet | Sun, Yiding Wang, Feng Zhu, Yutao Zhao, Wayne Xin Mao, Jiaxin |
| contents | The ability of the foundation models heavily relies on large-scale, diverse, and high-quality pretraining data. In order to improve data quality, researchers and practitioners often have to manually curate datasets from difference sources and develop dedicated data cleansing pipeline for each data repository. Lacking a unified data processing framework, this process is repetitive and cumbersome. To mitigate this issue, we propose a data processing framework that integrates a Processing Module which consists of a series of operators at different granularity levels, and an Analyzing Module which supports probing and evaluation of the refined data. The proposed framework is easy to use and highly flexible. In this demo paper, we first introduce how to use this framework with some example use cases and then demonstrate its effectiveness in improving the data quality with an automated evaluation with ChatGPT and an end-to-end evaluation in pretraining the GPT-2 model. The code and demonstration videos are accessible on GitHub. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_16358 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | An Integrated Data Processing Framework for Pretraining Foundation Models Sun, Yiding Wang, Feng Zhu, Yutao Zhao, Wayne Xin Mao, Jiaxin Machine Learning Computation and Language Information Retrieval The ability of the foundation models heavily relies on large-scale, diverse, and high-quality pretraining data. In order to improve data quality, researchers and practitioners often have to manually curate datasets from difference sources and develop dedicated data cleansing pipeline for each data repository. Lacking a unified data processing framework, this process is repetitive and cumbersome. To mitigate this issue, we propose a data processing framework that integrates a Processing Module which consists of a series of operators at different granularity levels, and an Analyzing Module which supports probing and evaluation of the refined data. The proposed framework is easy to use and highly flexible. In this demo paper, we first introduce how to use this framework with some example use cases and then demonstrate its effectiveness in improving the data quality with an automated evaluation with ChatGPT and an end-to-end evaluation in pretraining the GPT-2 model. The code and demonstration videos are accessible on GitHub. |
| title | An Integrated Data Processing Framework for Pretraining Foundation Models |
| topic | Machine Learning Computation and Language Information Retrieval |
| url | https://arxiv.org/abs/2402.16358 |