Beyond Scaling: Measuring and Predicting the Upper Bound of Knowledge Retention in Language Model Pre-Training
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866908750364803072 |
|---|---|
| author | Jiang, Changhao Zhang, Ming Cao, Yifei Ye, Junjie Fan, Xiaoran Dou, Shihan Xi, Zhiheng Sun, Jiajun Dong, Yi Shen, Yujiong Tong, Jingqi Fan, Baoyu Gui, Tao Zhang, Qi Huang, Xuanjing |
| author_facet | Jiang, Changhao Zhang, Ming Cao, Yifei Ye, Junjie Fan, Xiaoran Dou, Shihan Xi, Zhiheng Sun, Jiajun Dong, Yi Shen, Yujiong Tong, Jingqi Fan, Baoyu Gui, Tao Zhang, Qi Huang, Xuanjing |
| contents | The GPT-4 technical report suggests that downstream performance can be predicted from pre-training signals, but offers little methodological detail on how to quantify this. This work address this gap by modeling knowledge retention, the capacity of a pre-trained language model to memorize factual information from its corpus, and introduce a principled method to estimate it prior to training. We propose Size-dependent Mutual Information (SMI), an information-theoretic predictor that integrates knowledge frequency, knowledge specificity, and model size to forecast closed-book question answering (QA) accuracy. SMI is validated through large-scale document retrieval over the disclosed pre-training corpora of 21 public and 3 custom models, combined with a robust multi-template QA evaluation. Experiments show that SMI significantly outperforms repetition-based baselines and achieves $R^2$ > 0.7 in predicting QA accuracy for models above 1B parameters, without additional training. The analysis further reveals diminishing returns from scaling data and model size and provides evidence for an intrinsic upper bound on knowledge retention achievable by pre-training alone, motivating retrieval and other augmentation strategies. The dataset and code are available at https://github.com/yuhui1038/SMI. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_04066 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Beyond Scaling: Measuring and Predicting the Upper Bound of Knowledge Retention in Language Model Pre-Training Jiang, Changhao Zhang, Ming Cao, Yifei Ye, Junjie Fan, Xiaoran Dou, Shihan Xi, Zhiheng Sun, Jiajun Dong, Yi Shen, Yujiong Tong, Jingqi Fan, Baoyu Gui, Tao Zhang, Qi Huang, Xuanjing Computation and Language Artificial Intelligence The GPT-4 technical report suggests that downstream performance can be predicted from pre-training signals, but offers little methodological detail on how to quantify this. This work address this gap by modeling knowledge retention, the capacity of a pre-trained language model to memorize factual information from its corpus, and introduce a principled method to estimate it prior to training. We propose Size-dependent Mutual Information (SMI), an information-theoretic predictor that integrates knowledge frequency, knowledge specificity, and model size to forecast closed-book question answering (QA) accuracy. SMI is validated through large-scale document retrieval over the disclosed pre-training corpora of 21 public and 3 custom models, combined with a robust multi-template QA evaluation. Experiments show that SMI significantly outperforms repetition-based baselines and achieves $R^2$ > 0.7 in predicting QA accuracy for models above 1B parameters, without additional training. The analysis further reveals diminishing returns from scaling data and model size and provides evidence for an intrinsic upper bound on knowledge retention achievable by pre-training alone, motivating retrieval and other augmentation strategies. The dataset and code are available at https://github.com/yuhui1038/SMI. |
| title | Beyond Scaling: Measuring and Predicting the Upper Bound of Knowledge Retention in Language Model Pre-Training |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2502.04066 |