Enregistré dans:
| Auteurs principaux: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2026
|
| Sujets: | |
| Accès en ligne: | https://arxiv.org/abs/2605.02364 |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866909011768508416 |
|---|---|
| author | Liu, Fengze Zhou, Weidong Liu, Binbin Guo, Ping Wang, Zijun Zhang, Bingni Zhang, Yifan Yu, Yifeng Zhou, Xiaohuan Wang, Taifeng |
| author_facet | Liu, Fengze Zhou, Weidong Liu, Binbin Guo, Ping Wang, Zijun Zhang, Bingni Zhang, Yifan Yu, Yifeng Zhou, Xiaohuan Wang, Taifeng |
| contents | Upweighting high-quality data in LLM pretraining often improves performance, but in datalimited regimes, especially under overtraining, stronger upweighting increases repetition and can degrade performance. However, standard scaling laws do not reliably extrapolate across mixture recipes or under repetitions, making the selection for optimal data recipes at scaling underdetermined. To solve this, we introduce InfoLaw (Information Scaling Laws), a data-aware scaling framework that predicts loss from consumed tokens, model size, data mixture weights, and repetition. The key idea is to model pretraining as information accumulation, where quality controls information density and repetition induces scaledependent diminishing returns. We first collect the model performance after training on datasets that vary in scale, quality distribution, and repetition level. Then we build up the modeling for information so that information accurately predicts those model performance. InfoLaw predicts performance on unseen data recipes and larger scale runs (up to 7B, 425B tokens) with 0.15% mean and 0.96% max absolute error in loss, and it extrapolates reliably across overtraining levels, enabling efficient data-recipe selection under varying compute budgets. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_02364 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | InfoLaw: Information Scaling Laws for Large Language Models with Quality-Weighted Mixture Data and Repetition Liu, Fengze Zhou, Weidong Liu, Binbin Guo, Ping Wang, Zijun Zhang, Bingni Zhang, Yifan Yu, Yifeng Zhou, Xiaohuan Wang, Taifeng Computation and Language Upweighting high-quality data in LLM pretraining often improves performance, but in datalimited regimes, especially under overtraining, stronger upweighting increases repetition and can degrade performance. However, standard scaling laws do not reliably extrapolate across mixture recipes or under repetitions, making the selection for optimal data recipes at scaling underdetermined. To solve this, we introduce InfoLaw (Information Scaling Laws), a data-aware scaling framework that predicts loss from consumed tokens, model size, data mixture weights, and repetition. The key idea is to model pretraining as information accumulation, where quality controls information density and repetition induces scaledependent diminishing returns. We first collect the model performance after training on datasets that vary in scale, quality distribution, and repetition level. Then we build up the modeling for information so that information accurately predicts those model performance. InfoLaw predicts performance on unseen data recipes and larger scale runs (up to 7B, 425B tokens) with 0.15% mean and 0.96% max absolute error in loss, and it extrapolates reliably across overtraining levels, enabling efficient data-recipe selection under varying compute budgets. |
| title | InfoLaw: Information Scaling Laws for Large Language Models with Quality-Weighted Mixture Data and Repetition |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2605.02364 |