Deciphering the Impact of Pretraining Data on Large Language Models through Machine Unlearning
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866917760419758080 |
|---|---|
| author | Zhao, Yang Du, Li Ding, Xiao Xiong, Kai Sun, Zhouhao Shi, Jun Liu, Ting Qin, Bing |
| author_facet | Zhao, Yang Du, Li Ding, Xiao Xiong, Kai Sun, Zhouhao Shi, Jun Liu, Ting Qin, Bing |
| contents | Through pretraining on a corpus with various sources, Large Language Models (LLMs) have gained impressive performance. However, the impact of each component of the pretraining corpus remains opaque. As a result, the organization of the pretraining corpus is still empirical and may deviate from the optimal. To address this issue, we systematically analyze the impact of 48 datasets from 5 major categories of pretraining data of LLMs and measure their impacts on LLMs using benchmarks about nine major categories of model capabilities. Our analyses provide empirical results about the contribution of multiple corpora on the performances of LLMs, along with their joint impact patterns, including complementary, orthogonal, and correlational relationships. We also identify a set of ``high-impact data'' such as Books that is significantly related to a set of model capabilities. These findings provide insights into the organization of data to support more efficient pretraining of LLMs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_11537 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Deciphering the Impact of Pretraining Data on Large Language Models through Machine Unlearning Zhao, Yang Du, Li Ding, Xiao Xiong, Kai Sun, Zhouhao Shi, Jun Liu, Ting Qin, Bing Computation and Language Artificial Intelligence Through pretraining on a corpus with various sources, Large Language Models (LLMs) have gained impressive performance. However, the impact of each component of the pretraining corpus remains opaque. As a result, the organization of the pretraining corpus is still empirical and may deviate from the optimal. To address this issue, we systematically analyze the impact of 48 datasets from 5 major categories of pretraining data of LLMs and measure their impacts on LLMs using benchmarks about nine major categories of model capabilities. Our analyses provide empirical results about the contribution of multiple corpora on the performances of LLMs, along with their joint impact patterns, including complementary, orthogonal, and correlational relationships. We also identify a set of ``high-impact data'' such as Books that is significantly related to a set of model capabilities. These findings provide insights into the organization of data to support more efficient pretraining of LLMs. |
| title | Deciphering the Impact of Pretraining Data on Large Language Models through Machine Unlearning |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2402.11537 |