Spend Less, Fit Better: Budget-Efficient Scaling Law Fitting via Active Experiment Selection
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2026
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866908991227953152 |
|---|---|
| author | Li, Sijie Li, Shanda Lin, Haowei Sun, Weiwei Talwalkar, Ameet Yang, Yiming |
| author_facet | Li, Sijie Li, Shanda Lin, Haowei Sun, Weiwei Talwalkar, Ameet Yang, Yiming |
| contents | Scaling laws are used to plan multi-million-dollar training runs, but fitting those laws can itself cost millions. In modern large-scale workflows, assembling a sufficiently informative set of pilot experiments is already a major budget-allocation problem rather than a routine preprocessing step. We formulate scaling-law fitting as budget-aware sequential experimental design: given a finite pool of runnable experiments with heterogeneous costs, choose which runs to execute so as to maximize extrapolation accuracy in a high-cost target region. We then propose an uncertainty-aware method for sequentially allocating experimental budget toward the runs most useful for target-region extrapolation. Across a diverse benchmark of scaling-law tasks, our method consistently outperforms classical design-based baselines, and often approaches the performance of fitting on the full experimental set while using only about 10% of the total training budget. Our code is available at https://github.com/PlanarG/active-sl. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_22753 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Spend Less, Fit Better: Budget-Efficient Scaling Law Fitting via Active Experiment Selection Li, Sijie Li, Shanda Lin, Haowei Sun, Weiwei Talwalkar, Ameet Yang, Yiming Machine Learning Scaling laws are used to plan multi-million-dollar training runs, but fitting those laws can itself cost millions. In modern large-scale workflows, assembling a sufficiently informative set of pilot experiments is already a major budget-allocation problem rather than a routine preprocessing step. We formulate scaling-law fitting as budget-aware sequential experimental design: given a finite pool of runnable experiments with heterogeneous costs, choose which runs to execute so as to maximize extrapolation accuracy in a high-cost target region. We then propose an uncertainty-aware method for sequentially allocating experimental budget toward the runs most useful for target-region extrapolation. Across a diverse benchmark of scaling-law tasks, our method consistently outperforms classical design-based baselines, and often approaches the performance of fitting on the full experimental set while using only about 10% of the total training budget. Our code is available at https://github.com/PlanarG/active-sl. |
| title | Spend Less, Fit Better: Budget-Efficient Scaling Law Fitting via Active Experiment Selection |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2604.22753 |