Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2410.12896 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908093826203648 |
|---|---|
| author | Wang, Ke Zhu, Jiahui Ren, Minjie Liu, Zeming Li, Shiwei Zhang, Zongye Zhang, Chenkai Wu, Xiaoyu Zhan, Qiqi Liu, Qingjie Wang, Yunhong |
| author_facet | Wang, Ke Zhu, Jiahui Ren, Minjie Liu, Zeming Li, Shiwei Zhang, Zongye Zhang, Chenkai Wu, Xiaoyu Zhan, Qiqi Liu, Qingjie Wang, Yunhong |
| contents | The success of Large Language Models (LLMs) is inherently linked to the availability of vast, diverse, and high-quality data for training and evaluation. However, the growth rate of high-quality data is significantly outpaced by the expansion of training datasets, leading to a looming data exhaustion crisis. This underscores the urgent need to enhance data efficiency and explore new data sources. In this context, synthetic data has emerged as a promising solution. Currently, data generation primarily consists of two major approaches: data augmentation and synthesis. This paper comprehensively reviews and summarizes data generation techniques throughout the lifecycle of LLMs, including data preparation, pre-training, fine-tuning, instruction-tuning, preference alignment, and applications. Furthermore, We discuss the current constraints faced by these methods and investigate potential pathways for future development and research. Our aspiration is to equip researchers with a clear understanding of these methodologies, enabling them to swiftly identify appropriate data generation strategies in the construction of LLMs, while providing valuable insights for future exploration. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_12896 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | A Survey on Data Synthesis and Augmentation for Large Language Models Wang, Ke Zhu, Jiahui Ren, Minjie Liu, Zeming Li, Shiwei Zhang, Zongye Zhang, Chenkai Wu, Xiaoyu Zhan, Qiqi Liu, Qingjie Wang, Yunhong Computation and Language The success of Large Language Models (LLMs) is inherently linked to the availability of vast, diverse, and high-quality data for training and evaluation. However, the growth rate of high-quality data is significantly outpaced by the expansion of training datasets, leading to a looming data exhaustion crisis. This underscores the urgent need to enhance data efficiency and explore new data sources. In this context, synthetic data has emerged as a promising solution. Currently, data generation primarily consists of two major approaches: data augmentation and synthesis. This paper comprehensively reviews and summarizes data generation techniques throughout the lifecycle of LLMs, including data preparation, pre-training, fine-tuning, instruction-tuning, preference alignment, and applications. Furthermore, We discuss the current constraints faced by these methods and investigate potential pathways for future development and research. Our aspiration is to equip researchers with a clear understanding of these methodologies, enabling them to swiftly identify appropriate data generation strategies in the construction of LLMs, while providing valuable insights for future exploration. |
| title | A Survey on Data Synthesis and Augmentation for Large Language Models |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2410.12896 |