The Data-Quality Illusion: Rethinking Classifier-Based Quality Filtering for LLM Pretraining
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866908572767485952 |
|---|---|
| author | Saada, Thiziri Nait Bethune, Louis Klein, Michal Grangier, David Cuturi, Marco Ablin, Pierre |
| author_facet | Saada, Thiziri Nait Bethune, Louis Klein, Michal Grangier, David Cuturi, Marco Ablin, Pierre |
| contents | Large-scale models are pretrained on massive web-crawled datasets containing documents of mixed quality, making data filtering essential. A popular method is Classifier-based Quality Filtering (CQF), which trains a binary classifier to distinguish between pretraining data and a small, high-quality set. It assigns each pretraining document a quality score defined as the classifier's score and retains only the top-scoring ones. We provide an in-depth analysis of CQF. We show that while CQF improves downstream task performance, it does not necessarily enhance language modeling on the high-quality dataset. We explain this paradox by the fact that CQF implicitly filters the high-quality dataset as well. We further compare the behavior of models trained with CQF to those trained on synthetic data of increasing quality, obtained via random token permutations, and find starkly different trends. Our results challenge the view that CQF captures a meaningful notion of data quality. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_00866 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | The Data-Quality Illusion: Rethinking Classifier-Based Quality Filtering for LLM Pretraining Saada, Thiziri Nait Bethune, Louis Klein, Michal Grangier, David Cuturi, Marco Ablin, Pierre Machine Learning Computation and Language Large-scale models are pretrained on massive web-crawled datasets containing documents of mixed quality, making data filtering essential. A popular method is Classifier-based Quality Filtering (CQF), which trains a binary classifier to distinguish between pretraining data and a small, high-quality set. It assigns each pretraining document a quality score defined as the classifier's score and retains only the top-scoring ones. We provide an in-depth analysis of CQF. We show that while CQF improves downstream task performance, it does not necessarily enhance language modeling on the high-quality dataset. We explain this paradox by the fact that CQF implicitly filters the high-quality dataset as well. We further compare the behavior of models trained with CQF to those trained on synthetic data of increasing quality, obtained via random token permutations, and find starkly different trends. Our results challenge the view that CQF captures a meaningful notion of data quality. |
| title | The Data-Quality Illusion: Rethinking Classifier-Based Quality Filtering for LLM Pretraining |
| topic | Machine Learning Computation and Language |
| url | https://arxiv.org/abs/2510.00866 |