Data Proportion Detection for Optimized Data Management for Large Language Models
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866916411652177920 |
|---|---|
| author | Liang, Hao Zhao, Keshi Yang, Yajie Cui, Bin Dong, Guosheng Zhou, Zenan Zhang, Wentao |
| author_facet | Liang, Hao Zhao, Keshi Yang, Yajie Cui, Bin Dong, Guosheng Zhou, Zenan Zhang, Wentao |
| contents | Large language models (LLMs) have demonstrated exceptional performance across a wide range of tasks and domains, with data preparation playing a critical role in achieving these results. Pre-training data typically combines information from multiple domains. To maximize performance when integrating data from various domains, determining the optimal data proportion is essential. However, state-of-the-art (SOTA) LLMs rarely disclose details about their pre-training data, making it difficult for researchers to identify ideal data proportions. In this paper, we introduce a new topic, \textit{data proportion detection}, which enables the automatic estimation of pre-training data proportions by analyzing the generated outputs of LLMs. We provide rigorous theoretical proofs, practical algorithms, and preliminary experimental results for data proportion detection. Based on these findings, we offer valuable insights into the challenges and future directions for effective data proportion detection and data management. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2409_17527 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Data Proportion Detection for Optimized Data Management for Large Language Models Liang, Hao Zhao, Keshi Yang, Yajie Cui, Bin Dong, Guosheng Zhou, Zenan Zhang, Wentao Computation and Language Large language models (LLMs) have demonstrated exceptional performance across a wide range of tasks and domains, with data preparation playing a critical role in achieving these results. Pre-training data typically combines information from multiple domains. To maximize performance when integrating data from various domains, determining the optimal data proportion is essential. However, state-of-the-art (SOTA) LLMs rarely disclose details about their pre-training data, making it difficult for researchers to identify ideal data proportions. In this paper, we introduce a new topic, \textit{data proportion detection}, which enables the automatic estimation of pre-training data proportions by analyzing the generated outputs of LLMs. We provide rigorous theoretical proofs, practical algorithms, and preliminary experimental results for data proportion detection. Based on these findings, we offer valuable insights into the challenges and future directions for effective data proportion detection and data management. |
| title | Data Proportion Detection for Optimized Data Management for Large Language Models |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2409.17527 |