Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2601.12343 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911383820435456 |
|---|---|
| author | Gao, Wayne Han, Sukjin Liang, Annie |
| author_facet | Gao, Wayne Han, Sukjin Liang, Annie |
| contents | Large language models (LLMs) are increasingly used to predict human behavior. We propose a measure for evaluating how much knowledge a pretrained LLM brings to such a prediction: its equivalent sample size, defined as the amount of task-specific data needed to match the predictive accuracy of the LLM. We estimate this measure by comparing the prediction error of a fixed LLM in a given domain to that of flexible machine learning models trained on increasing samples of domain-specific data. We further provide a statistical inference procedure by developing a new asymptotic theory for cross-validated prediction error. Finally, we apply this method to the Panel Study of Income Dynamics. We find that LLMs encode considerable predictive information for some economic variables but much less for others, suggesting that their value as substitutes for domain-specific data differs markedly across settings. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_12343 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | How Well Do LLMs Predict Human Behavior? A Measure of their Pretrained Knowledge Gao, Wayne Han, Sukjin Liang, Annie Econometrics Artificial Intelligence Machine Learning Large language models (LLMs) are increasingly used to predict human behavior. We propose a measure for evaluating how much knowledge a pretrained LLM brings to such a prediction: its equivalent sample size, defined as the amount of task-specific data needed to match the predictive accuracy of the LLM. We estimate this measure by comparing the prediction error of a fixed LLM in a given domain to that of flexible machine learning models trained on increasing samples of domain-specific data. We further provide a statistical inference procedure by developing a new asymptotic theory for cross-validated prediction error. Finally, we apply this method to the Panel Study of Income Dynamics. We find that LLMs encode considerable predictive information for some economic variables but much less for others, suggesting that their value as substitutes for domain-specific data differs markedly across settings. |
| title | How Well Do LLMs Predict Human Behavior? A Measure of their Pretrained Knowledge |
| topic | Econometrics Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2601.12343 |