Non-Vacuous Generalization Bounds for Large Language Models
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2023
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866914873766576128 |
|---|---|
| author | Lotfi, Sanae Finzi, Marc Kuang, Yilun Rudner, Tim G. J. Goldblum, Micah Wilson, Andrew Gordon |
| author_facet | Lotfi, Sanae Finzi, Marc Kuang, Yilun Rudner, Tim G. J. Goldblum, Micah Wilson, Andrew Gordon |
| contents | Modern language models can contain billions of parameters, raising the question of whether they can generalize beyond the training data or simply parrot their training corpora. We provide the first non-vacuous generalization bounds for pretrained large language models (LLMs), indicating that language models are capable of discovering regularities that generalize to unseen data. In particular, we derive a compression bound that is valid for the unbounded log-likelihood loss using prediction smoothing, and we extend the bound to handle subsampling, accelerating bound computation by orders of magnitude on massive datasets. To achieve the extreme level of compression required for non-vacuous bounds, we devise SubLoRA, a simple low-dimensional nonlinear parameterization that leads to non-vacuous generalization bounds for models with nearly a billion parameters. Finally, we use our bounds to understand LLM generalization and find that larger models have better generalization bounds and are more compressible than smaller models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2312_17173 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Non-Vacuous Generalization Bounds for Large Language Models Lotfi, Sanae Finzi, Marc Kuang, Yilun Rudner, Tim G. J. Goldblum, Micah Wilson, Andrew Gordon Machine Learning Modern language models can contain billions of parameters, raising the question of whether they can generalize beyond the training data or simply parrot their training corpora. We provide the first non-vacuous generalization bounds for pretrained large language models (LLMs), indicating that language models are capable of discovering regularities that generalize to unseen data. In particular, we derive a compression bound that is valid for the unbounded log-likelihood loss using prediction smoothing, and we extend the bound to handle subsampling, accelerating bound computation by orders of magnitude on massive datasets. To achieve the extreme level of compression required for non-vacuous bounds, we devise SubLoRA, a simple low-dimensional nonlinear parameterization that leads to non-vacuous generalization bounds for models with nearly a billion parameters. Finally, we use our bounds to understand LLM generalization and find that larger models have better generalization bounds and are more compressible than smaller models. |
| title | Non-Vacuous Generalization Bounds for Large Language Models |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2312.17173 |