Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2604.05364 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908941695320064 |
|---|---|
| author | Ahamed, Md Atik Parmar, Mihir Goyal, Palash Song, Yiwen Le, Long T. Cheng, Qiang Li, Chun-Liang Palangi, Hamid Yoon, Jinsung Pfister, Tomas |
| author_facet | Ahamed, Md Atik Parmar, Mihir Goyal, Palash Song, Yiwen Le, Long T. Cheng, Qiang Li, Chun-Liang Palangi, Hamid Yoon, Jinsung Pfister, Tomas |
| contents | We introduce TFRBench, the first benchmark designed to evaluate the reasoning capabilities of forecasting systems. Traditionally, time-series forecasting has been evaluated solely on numerical accuracy, treating foundation models as ``black boxes.'' Unlike existing benchmarks, TFRBench provides a protocol for evaluating the reasoning generated by forecasting systems--specifically their analysis of cross-channel dependencies, trends, and external events. To enable this, we propose a systematic multi-agent framework that utilizes an iterative verification loop to synthesize numerically grounded reasoning traces. Spanning ten datasets across five domains, our evaluation confirms that this reasoning is causally effective; useful for evaluation; and prompting LLMs with our generated traces significantly improves forecasting accuracy compared to direct numerical prediction (e.g., avg. $\sim40.2\%\to56.6\%)$, validating the quality of our reasoning. Conversely, benchmarking experiments reveal that off-the-shelf LLMs consistently struggle with both reasoning (lower LLM-as-a-Judge scores) and numerical forecasting, frequently failing to capture domain-specific dynamics. TFRBench thus establishes a new standard for interpretable, reasoning-based evaluation in time-series forecasting. Our benchmark is available at: https://tfrbench.github.io |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_05364 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | TFRBench: A Reasoning Benchmark for Evaluating Forecasting Systems Ahamed, Md Atik Parmar, Mihir Goyal, Palash Song, Yiwen Le, Long T. Cheng, Qiang Li, Chun-Liang Palangi, Hamid Yoon, Jinsung Pfister, Tomas Artificial Intelligence We introduce TFRBench, the first benchmark designed to evaluate the reasoning capabilities of forecasting systems. Traditionally, time-series forecasting has been evaluated solely on numerical accuracy, treating foundation models as ``black boxes.'' Unlike existing benchmarks, TFRBench provides a protocol for evaluating the reasoning generated by forecasting systems--specifically their analysis of cross-channel dependencies, trends, and external events. To enable this, we propose a systematic multi-agent framework that utilizes an iterative verification loop to synthesize numerically grounded reasoning traces. Spanning ten datasets across five domains, our evaluation confirms that this reasoning is causally effective; useful for evaluation; and prompting LLMs with our generated traces significantly improves forecasting accuracy compared to direct numerical prediction (e.g., avg. $\sim40.2\%\to56.6\%)$, validating the quality of our reasoning. Conversely, benchmarking experiments reveal that off-the-shelf LLMs consistently struggle with both reasoning (lower LLM-as-a-Judge scores) and numerical forecasting, frequently failing to capture domain-specific dynamics. TFRBench thus establishes a new standard for interpretable, reasoning-based evaluation in time-series forecasting. Our benchmark is available at: https://tfrbench.github.io |
| title | TFRBench: A Reasoning Benchmark for Evaluating Forecasting Systems |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2604.05364 |