Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2501.11747 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916581306531840 |
|---|---|
| author | Held, William Paranjape, Bhargavi Koura, Punit Singh Lewis, Mike Zhang, Frank Mihaylov, Todor |
| author_facet | Held, William Paranjape, Bhargavi Koura, Punit Singh Lewis, Mike Zhang, Frank Mihaylov, Todor |
| contents | Large Language Models improve with increasing amounts of high-quality training data. However, leveraging larger datasets requires balancing quality, quantity, and diversity across sources. After evaluating nine baseline methods under both compute- and data-constrained scenarios, we find token-count heuristics outperform manual and learned mixes, indicating that simple approaches accounting for dataset size and diversity are surprisingly effective. Building on this insight, we propose two complementary approaches: UtiliMax, which extends token-based heuristics by incorporating utility estimates from reduced-scale ablations, achieving up to a 10.6x speedup over manual baselines; and Model Estimated Data Utility (MEDU), which leverages LLMs to estimate data utility from small samples, matching ablation-based performance while reducing computational requirements by $\sim$200x. Together, these approaches establish a new framework for automated, compute-efficient data mixing that is robust across training regimes. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_11747 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Optimizing Pretraining Data Mixtures with LLM-Estimated Utility Held, William Paranjape, Bhargavi Koura, Punit Singh Lewis, Mike Zhang, Frank Mihaylov, Todor Computation and Language Artificial Intelligence Large Language Models improve with increasing amounts of high-quality training data. However, leveraging larger datasets requires balancing quality, quantity, and diversity across sources. After evaluating nine baseline methods under both compute- and data-constrained scenarios, we find token-count heuristics outperform manual and learned mixes, indicating that simple approaches accounting for dataset size and diversity are surprisingly effective. Building on this insight, we propose two complementary approaches: UtiliMax, which extends token-based heuristics by incorporating utility estimates from reduced-scale ablations, achieving up to a 10.6x speedup over manual baselines; and Model Estimated Data Utility (MEDU), which leverages LLMs to estimate data utility from small samples, matching ablation-based performance while reducing computational requirements by $\sim$200x. Together, these approaches establish a new framework for automated, compute-efficient data mixing that is robust across training regimes. |
| title | Optimizing Pretraining Data Mixtures with LLM-Estimated Utility |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2501.11747 |