Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2504.14765 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912763090042880 |
|---|---|
| author | Lopez-Lira, Alejandro Tang, Yuehua Zhu, Mingyin |
| author_facet | Lopez-Lira, Alejandro Tang, Yuehua Zhu, Mingyin |
| contents | Large language models (LLMs) cannot be trusted for economic forecasts during periods covered by their training data. Counterfactual forecasting ability is non-identified when the model has seen the realized values: any observed output is consistent with both genuine skill and memorization. Any evidence of memorization represents only a lower bound on encoded knowledge. We demonstrate LLMs have memorized economic and financial data, recalling exact values before their knowledge cutoff. Instructions to respect historical boundaries fail to prevent recall-level accuracy, and masking fails as LLMs reconstruct entities and dates from minimal context. Post-cutoff, we observe no recall. Memorization extends to embeddings. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_14765 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | The Memorization Problem: Can We Trust LLMs' Economic Forecasts? Lopez-Lira, Alejandro Tang, Yuehua Zhu, Mingyin General Finance Statistical Finance Large language models (LLMs) cannot be trusted for economic forecasts during periods covered by their training data. Counterfactual forecasting ability is non-identified when the model has seen the realized values: any observed output is consistent with both genuine skill and memorization. Any evidence of memorization represents only a lower bound on encoded knowledge. We demonstrate LLMs have memorized economic and financial data, recalling exact values before their knowledge cutoff. Instructions to respect historical boundaries fail to prevent recall-level accuracy, and masking fails as LLMs reconstruct entities and dates from minimal context. Post-cutoff, we observe no recall. Memorization extends to embeddings. |
| title | The Memorization Problem: Can We Trust LLMs' Economic Forecasts? |
| topic | General Finance Statistical Finance |
| url | https://arxiv.org/abs/2504.14765 |