A Survey on Data Selection for Language Models
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866910551431446528 |
|---|---|
| author | Albalak, Alon Elazar, Yanai Xie, Sang Michael Longpre, Shayne Lambert, Nathan Wang, Xinyi Muennighoff, Niklas Hou, Bairu Pan, Liangming Jeong, Haewon Raffel, Colin Chang, Shiyu Hashimoto, Tatsunori Wang, William Yang |
| author_facet | Albalak, Alon Elazar, Yanai Xie, Sang Michael Longpre, Shayne Lambert, Nathan Wang, Xinyi Muennighoff, Niklas Hou, Bairu Pan, Liangming Jeong, Haewon Raffel, Colin Chang, Shiyu Hashimoto, Tatsunori Wang, William Yang |
| contents | A major factor in the recent success of large language models is the use of enormous and ever-growing text datasets for unsupervised pre-training. However, naively training a model on all available data may not be optimal (or feasible), as the quality of available text data can vary. Filtering out data can also decrease the carbon footprint and financial costs of training models by reducing the amount of training required. Data selection methods aim to determine which candidate data points to include in the training dataset and how to appropriately sample from the selected data points. The promise of improved data selection methods has caused the volume of research in the area to rapidly expand. However, because deep learning is mostly driven by empirical evidence and experimentation on large-scale data is expensive, few organizations have the resources for extensive data selection research. Consequently, knowledge of effective data selection practices has become concentrated within a few organizations, many of which do not openly share their findings and methodologies. To narrow this gap in knowledge, we present a comprehensive review of existing literature on data selection methods and related research areas, providing a taxonomy of existing approaches. By describing the current landscape of research, this work aims to accelerate progress in data selection by establishing an entry point for new and established researchers. Additionally, throughout this review we draw attention to noticeable holes in the literature and conclude the paper by proposing promising avenues for future research. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_16827 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | A Survey on Data Selection for Language Models Albalak, Alon Elazar, Yanai Xie, Sang Michael Longpre, Shayne Lambert, Nathan Wang, Xinyi Muennighoff, Niklas Hou, Bairu Pan, Liangming Jeong, Haewon Raffel, Colin Chang, Shiyu Hashimoto, Tatsunori Wang, William Yang Computation and Language Machine Learning A major factor in the recent success of large language models is the use of enormous and ever-growing text datasets for unsupervised pre-training. However, naively training a model on all available data may not be optimal (or feasible), as the quality of available text data can vary. Filtering out data can also decrease the carbon footprint and financial costs of training models by reducing the amount of training required. Data selection methods aim to determine which candidate data points to include in the training dataset and how to appropriately sample from the selected data points. The promise of improved data selection methods has caused the volume of research in the area to rapidly expand. However, because deep learning is mostly driven by empirical evidence and experimentation on large-scale data is expensive, few organizations have the resources for extensive data selection research. Consequently, knowledge of effective data selection practices has become concentrated within a few organizations, many of which do not openly share their findings and methodologies. To narrow this gap in knowledge, we present a comprehensive review of existing literature on data selection methods and related research areas, providing a taxonomy of existing approaches. By describing the current landscape of research, this work aims to accelerate progress in data selection by establishing an entry point for new and established researchers. Additionally, throughout this review we draw attention to noticeable holes in the literature and conclude the paper by proposing promising avenues for future research. |
| title | A Survey on Data Selection for Language Models |
| topic | Computation and Language Machine Learning |
| url | https://arxiv.org/abs/2402.16827 |