Improving the Scaling Laws of Synthetic Data with Deliberate Practice
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866909504260538368 |
|---|---|
| author | Askari-Hemmat, Reyhane Pezeshki, Mohammad Dohmatob, Elvis Bordes, Florian Astolfi, Pietro Hall, Melissa Verbeek, Jakob Drozdzal, Michal Romero-Soriano, Adriana |
| author_facet | Askari-Hemmat, Reyhane Pezeshki, Mohammad Dohmatob, Elvis Bordes, Florian Astolfi, Pietro Hall, Melissa Verbeek, Jakob Drozdzal, Michal Romero-Soriano, Adriana |
| contents | Inspired by the principle of deliberate practice in human learning, we propose Deliberate Practice for Synthetic Data Generation (DP), a novel framework that improves sample efficiency through dynamic synthetic data generation. Prior work has shown that scaling synthetic data is inherently challenging, as naively adding new data leads to diminishing returns. To address this, pruning has been identified as a key mechanism for improving scaling, enabling models to focus on the most informative synthetic samples. Rather than generating a large dataset and pruning it afterward, DP efficiently approximates the direct generation of informative samples. We theoretically show how training on challenging, informative examples improves scaling laws and empirically validate that DP achieves better scaling performance with significantly fewer training samples and iterations. On ImageNet-100, DP generates 3.4x fewer samples and requires six times fewer iterations, while on ImageNet-1k, it generates 8x fewer samples with a 30 percent reduction in iterations, all while achieving superior performance compared to prior work. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_15588 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Improving the Scaling Laws of Synthetic Data with Deliberate Practice Askari-Hemmat, Reyhane Pezeshki, Mohammad Dohmatob, Elvis Bordes, Florian Astolfi, Pietro Hall, Melissa Verbeek, Jakob Drozdzal, Michal Romero-Soriano, Adriana Machine Learning Artificial Intelligence Inspired by the principle of deliberate practice in human learning, we propose Deliberate Practice for Synthetic Data Generation (DP), a novel framework that improves sample efficiency through dynamic synthetic data generation. Prior work has shown that scaling synthetic data is inherently challenging, as naively adding new data leads to diminishing returns. To address this, pruning has been identified as a key mechanism for improving scaling, enabling models to focus on the most informative synthetic samples. Rather than generating a large dataset and pruning it afterward, DP efficiently approximates the direct generation of informative samples. We theoretically show how training on challenging, informative examples improves scaling laws and empirically validate that DP achieves better scaling performance with significantly fewer training samples and iterations. On ImageNet-100, DP generates 3.4x fewer samples and requires six times fewer iterations, while on ImageNet-1k, it generates 8x fewer samples with a 30 percent reduction in iterations, all while achieving superior performance compared to prior work. |
| title | Improving the Scaling Laws of Synthetic Data with Deliberate Practice |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2502.15588 |