Saved in:
| Main Authors: | , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2604.11529 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915940174659584 |
|---|---|
| author | Goktas, Denizalp Riaño-Briceño, Gerardo Abdullah, Alif Nair, Aryan Shen, Chenkai de Lucio, Beatriz Magnusson, Alexandra Mashrur, Farhan Abdulla, Ahmed Sen, Shawrna Thippireddy, Mahitha Schwartz, Gregory Greenwald, Amy |
| author_facet | Goktas, Denizalp Riaño-Briceño, Gerardo Abdullah, Alif Nair, Aryan Shen, Chenkai de Lucio, Beatriz Magnusson, Alexandra Mashrur, Farhan Abdulla, Ahmed Sen, Shawrna Thippireddy, Mahitha Schwartz, Gregory Greenwald, Amy |
| contents | Foundation models have transformed natural language processing and computer vision, and a rapidly growing literature on time-series foundation models (TSFMs) seeks to replicate this success in forecasting. While recent open-source models demonstrate the promise of TSFMs, the field lacks a comprehensive and community-accepted model evaluation framework. We see at least four major issues impeding progress on the development of such a framework. First, existing evaluation frameworks comprise benchmark forecasting tasks derived from often outdated datasets (e.g., M3), many of which lack clear metadata and overlap with the corpora used to pre-train TSFMs. Second, these frameworks evaluate models along a narrowly defined set of benchmark forecasting tasks, such as forecast horizon length or domain, but overlook core statistical properties such as non-stationarity and seasonality. Third, domain-specific models (e.g., XGBoost) are often compared unfairly, as existing frameworks do not enforce a systematic and consistent hyperparameter tuning convention for all models. Fourth, visualization tools for interpreting comparative performance are lacking. To address these issues, we introduce TempusBench, an open-source evaluation framework for TSFMs. TempusBench consists of 1) new datasets which are not included in existing TSFM pretraining corpora, 2) a set of novel benchmark tasks that go beyond existing ones, 3) a model evaluation pipeline with a standardized hyperparameter tuning protocol, and 4) a tensorboard-based visualization interface. We provide access to our code on GitHub: https://github.com/Smlcrm/TempusBench and maintain a live leaderboard at https://benchmark.smlcrm.com/. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_11529 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | TempusBench: An Evaluation Framework for Time-Series Forecasting Goktas, Denizalp Riaño-Briceño, Gerardo Abdullah, Alif Nair, Aryan Shen, Chenkai de Lucio, Beatriz Magnusson, Alexandra Mashrur, Farhan Abdulla, Ahmed Sen, Shawrna Thippireddy, Mahitha Schwartz, Gregory Greenwald, Amy Machine Learning Foundation models have transformed natural language processing and computer vision, and a rapidly growing literature on time-series foundation models (TSFMs) seeks to replicate this success in forecasting. While recent open-source models demonstrate the promise of TSFMs, the field lacks a comprehensive and community-accepted model evaluation framework. We see at least four major issues impeding progress on the development of such a framework. First, existing evaluation frameworks comprise benchmark forecasting tasks derived from often outdated datasets (e.g., M3), many of which lack clear metadata and overlap with the corpora used to pre-train TSFMs. Second, these frameworks evaluate models along a narrowly defined set of benchmark forecasting tasks, such as forecast horizon length or domain, but overlook core statistical properties such as non-stationarity and seasonality. Third, domain-specific models (e.g., XGBoost) are often compared unfairly, as existing frameworks do not enforce a systematic and consistent hyperparameter tuning convention for all models. Fourth, visualization tools for interpreting comparative performance are lacking. To address these issues, we introduce TempusBench, an open-source evaluation framework for TSFMs. TempusBench consists of 1) new datasets which are not included in existing TSFM pretraining corpora, 2) a set of novel benchmark tasks that go beyond existing ones, 3) a model evaluation pipeline with a standardized hyperparameter tuning protocol, and 4) a tensorboard-based visualization interface. We provide access to our code on GitHub: https://github.com/Smlcrm/TempusBench and maintain a live leaderboard at https://benchmark.smlcrm.com/. |
| title | TempusBench: An Evaluation Framework for Time-Series Forecasting |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2604.11529 |