A More Realistic Evaluation of Cross-Frequency Transfer Learning and Foundation Forecasting Models
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866912715942920192 |
|---|---|
| author | Olivares, Kin G. Wolff, Malcolm Konstantinova, Tatiana Ramasubramanian, Shankar Oreshkin, Boris Wilson, Andrew Gordon Potapczynski, Andres Potosnak, Willa Mahoney, Michael W. Cao, Mengfei Efimov, Dmitry |
| author_facet | Olivares, Kin G. Wolff, Malcolm Konstantinova, Tatiana Ramasubramanian, Shankar Oreshkin, Boris Wilson, Andrew Gordon Potapczynski, Andres Potosnak, Willa Mahoney, Michael W. Cao, Mengfei Efimov, Dmitry |
| contents | Cross-frequency transfer learning (CFTL) has emerged as a popular framework for curating large-scale time series datasets to pre-train foundation forecasting models (FFMs). Although CFTL has shown promise, current benchmarking practices fall short of accurately assessing its performance. This shortcoming stems from many factors: an over-reliance on small-scale evaluation datasets; inadequate treatment of sample size when computing summary statistics; reporting of suboptimal statistical models; and failing to account for non-negligible risks of overlap between pre-training and test datasets. To address these limitations, we introduce a unified reimplementation of widely-adopted neural forecasting networks, adapting them for the CFTL setup; we pre-train only on proprietary and synthetic data, being careful to prevent test leakage; and we evaluate on 15 large, diverse public forecast competition datasets. Our empirical analysis reveals that statistical models' accuracy is frequently underreported. Notably, we confirm that statistical models and their ensembles consistently outperform existing FFMs by more than 8.2% in sCRPS, and by more than 20% MASE, across datasets. However, we also find that synthetic dataset pre-training does improve the accuracy of a FFM by 7% percent. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_19465 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | A More Realistic Evaluation of Cross-Frequency Transfer Learning and Foundation Forecasting Models Olivares, Kin G. Wolff, Malcolm Konstantinova, Tatiana Ramasubramanian, Shankar Oreshkin, Boris Wilson, Andrew Gordon Potapczynski, Andres Potosnak, Willa Mahoney, Michael W. Cao, Mengfei Efimov, Dmitry Machine Learning Artificial Intelligence Applications Cross-frequency transfer learning (CFTL) has emerged as a popular framework for curating large-scale time series datasets to pre-train foundation forecasting models (FFMs). Although CFTL has shown promise, current benchmarking practices fall short of accurately assessing its performance. This shortcoming stems from many factors: an over-reliance on small-scale evaluation datasets; inadequate treatment of sample size when computing summary statistics; reporting of suboptimal statistical models; and failing to account for non-negligible risks of overlap between pre-training and test datasets. To address these limitations, we introduce a unified reimplementation of widely-adopted neural forecasting networks, adapting them for the CFTL setup; we pre-train only on proprietary and synthetic data, being careful to prevent test leakage; and we evaluate on 15 large, diverse public forecast competition datasets. Our empirical analysis reveals that statistical models' accuracy is frequently underreported. Notably, we confirm that statistical models and their ensembles consistently outperform existing FFMs by more than 8.2% in sCRPS, and by more than 20% MASE, across datasets. However, we also find that synthetic dataset pre-training does improve the accuracy of a FFM by 7% percent. |
| title | A More Realistic Evaluation of Cross-Frequency Transfer Learning and Foundation Forecasting Models |
| topic | Machine Learning Artificial Intelligence Applications |
| url | https://arxiv.org/abs/2509.19465 |