Perplexed by Perplexity: Perplexity-Based Data Pruning With Small Reference Models
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866929366942875648 |
|---|---|
| author | Ankner, Zachary Blakeney, Cody Sreenivasan, Kartik Marion, Max Leavitt, Matthew L. Paul, Mansheej |
| author_facet | Ankner, Zachary Blakeney, Cody Sreenivasan, Kartik Marion, Max Leavitt, Matthew L. Paul, Mansheej |
| contents | In this work, we investigate whether small language models can determine high-quality subsets of large-scale text datasets that improve the performance of larger language models. While existing work has shown that pruning based on the perplexity of a larger model can yield high-quality data, we investigate whether smaller models can be used for perplexity-based pruning and how pruning is affected by the domain composition of the data being pruned. We demonstrate that for multiple dataset compositions, perplexity-based pruning of pretraining data can \emph{significantly} improve downstream task performance: pruning based on perplexities computed with a 125 million parameter model improves the average performance on downstream tasks of a 3 billion parameter model by up to 2.04 and achieves up to a $1.45\times$ reduction in pretraining steps to reach commensurate baseline performance. Furthermore, we demonstrate that such perplexity-based data pruning also yields downstream performance gains in the over-trained and data-constrained regimes. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_20541 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Perplexed by Perplexity: Perplexity-Based Data Pruning With Small Reference Models Ankner, Zachary Blakeney, Cody Sreenivasan, Kartik Marion, Max Leavitt, Matthew L. Paul, Mansheej Machine Learning Computation and Language In this work, we investigate whether small language models can determine high-quality subsets of large-scale text datasets that improve the performance of larger language models. While existing work has shown that pruning based on the perplexity of a larger model can yield high-quality data, we investigate whether smaller models can be used for perplexity-based pruning and how pruning is affected by the domain composition of the data being pruned. We demonstrate that for multiple dataset compositions, perplexity-based pruning of pretraining data can \emph{significantly} improve downstream task performance: pruning based on perplexities computed with a 125 million parameter model improves the average performance on downstream tasks of a 3 billion parameter model by up to 2.04 and achieves up to a $1.45\times$ reduction in pretraining steps to reach commensurate baseline performance. Furthermore, we demonstrate that such perplexity-based data pruning also yields downstream performance gains in the over-trained and data-constrained regimes. |
| title | Perplexed by Perplexity: Perplexity-Based Data Pruning With Small Reference Models |
| topic | Machine Learning Computation and Language |
| url | https://arxiv.org/abs/2405.20541 |