Beyond Naïve Prompting: Strategies for Improved Context-aided Forecasting with LLMs
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866911472585539584 |
|---|---|
| author | Ashok, Arjun Williams, Andrew Robert Zheng, Vincent Zhihao Rish, Irina Chapados, Nicolas Marcotte, Étienne Zantedeschi, Valentina Drouin, Alexandre |
| author_facet | Ashok, Arjun Williams, Andrew Robert Zheng, Vincent Zhihao Rish, Irina Chapados, Nicolas Marcotte, Étienne Zantedeschi, Valentina Drouin, Alexandre |
| contents | Real-world forecasting requires models to integrate not only historical data but also relevant contextual information provided in textual form. While large language models (LLMs) show promise for context-aided forecasting, critical challenges remain: we lack diagnostic tools to understand failure modes, performance remains far below their potential, and high computational costs limit practical deployment. We introduce a unified framework of four strategies that address these limitations along three orthogonal dimensions: model diagnostics, accuracy, and efficiency. Through extensive evaluation across model families from small open-source models to frontier models including Gemini, GPT, and Claude, we uncover both fundamental insights and practical solutions. Our findings span three key dimensions: diagnostic strategies reveal the "Execution Gap" where models correctly explain how context affects forecasts but fail to apply this reasoning; accuracy-focused strategies achieve substantial performance improvements of 25-50%; and efficiency-oriented approaches show that adaptive routing between small and large models can approach large model accuracy on average while significantly reducing inference costs. These orthogonal strategies can be flexibly integrated based on deployment constraints, providing practitioners with a comprehensive toolkit for practical LLM-based context-aided forecasting. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_09904 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Beyond Naïve Prompting: Strategies for Improved Context-aided Forecasting with LLMs Ashok, Arjun Williams, Andrew Robert Zheng, Vincent Zhihao Rish, Irina Chapados, Nicolas Marcotte, Étienne Zantedeschi, Valentina Drouin, Alexandre Machine Learning Artificial Intelligence Real-world forecasting requires models to integrate not only historical data but also relevant contextual information provided in textual form. While large language models (LLMs) show promise for context-aided forecasting, critical challenges remain: we lack diagnostic tools to understand failure modes, performance remains far below their potential, and high computational costs limit practical deployment. We introduce a unified framework of four strategies that address these limitations along three orthogonal dimensions: model diagnostics, accuracy, and efficiency. Through extensive evaluation across model families from small open-source models to frontier models including Gemini, GPT, and Claude, we uncover both fundamental insights and practical solutions. Our findings span three key dimensions: diagnostic strategies reveal the "Execution Gap" where models correctly explain how context affects forecasts but fail to apply this reasoning; accuracy-focused strategies achieve substantial performance improvements of 25-50%; and efficiency-oriented approaches show that adaptive routing between small and large models can approach large model accuracy on average while significantly reducing inference costs. These orthogonal strategies can be flexibly integrated based on deployment constraints, providing practitioners with a comprehensive toolkit for practical LLM-based context-aided forecasting. |
| title | Beyond Naïve Prompting: Strategies for Improved Context-aided Forecasting with LLMs |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2508.09904 |