Compute-Optimal LLMs Provably Generalize Better With Scale
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909586603114496 |
|---|---|
| author | Finzi, Marc Kapoor, Sanyam Granziol, Diego Gu, Anming De Sa, Christopher Kolter, J. Zico Wilson, Andrew Gordon |
| author_facet | Finzi, Marc Kapoor, Sanyam Granziol, Diego Gu, Anming De Sa, Christopher Kolter, J. Zico Wilson, Andrew Gordon |
| contents | Why do larger language models generalize better? To investigate this question, we develop generalization bounds on the pretraining objective of large language models (LLMs) in the compute-optimal regime, as described by the Chinchilla scaling laws. We introduce a novel, fully empirical Freedman-type martingale concentration inequality that tightens existing bounds by accounting for the variance of the loss function. This generalization bound can be decomposed into three interpretable components: the number of parameters per token, the loss variance, and the quantization error at a fixed bitrate. As compute-optimal language models are scaled up, the number of parameters per data point remains constant; however, both the loss variance and the quantization error decrease, implying that larger models should have smaller generalization gaps. We examine why larger models tend to be more quantizable from an information theoretic perspective, showing that the rate at which they can integrate new information grows more slowly than their capacity on the compute-optimal frontier. From these findings we produce a scaling law for the generalization gap, with bounds that become predictably stronger with scale. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_15208 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Compute-Optimal LLMs Provably Generalize Better With Scale Finzi, Marc Kapoor, Sanyam Granziol, Diego Gu, Anming De Sa, Christopher Kolter, J. Zico Wilson, Andrew Gordon Machine Learning Artificial Intelligence Why do larger language models generalize better? To investigate this question, we develop generalization bounds on the pretraining objective of large language models (LLMs) in the compute-optimal regime, as described by the Chinchilla scaling laws. We introduce a novel, fully empirical Freedman-type martingale concentration inequality that tightens existing bounds by accounting for the variance of the loss function. This generalization bound can be decomposed into three interpretable components: the number of parameters per token, the loss variance, and the quantization error at a fixed bitrate. As compute-optimal language models are scaled up, the number of parameters per data point remains constant; however, both the loss variance and the quantization error decrease, implying that larger models should have smaller generalization gaps. We examine why larger models tend to be more quantizable from an information theoretic perspective, showing that the rate at which they can integrate new information grows more slowly than their capacity on the compute-optimal frontier. From these findings we produce a scaling law for the generalization gap, with bounds that become predictably stronger with scale. |
| title | Compute-Optimal LLMs Provably Generalize Better With Scale |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2504.15208 |