Random Initialization Can't Catch Up: The Advantage of Language Model Transfer for Time Series Forecasting
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866915361166721024 |
|---|---|
| author | Riachi, Roland Rasul, Kashif Ashok, Arjun Humane, Prateek Roger, Alexis Williams, Andrew R. Nevmyvaka, Yuriy Rish, Irina |
| author_facet | Riachi, Roland Rasul, Kashif Ashok, Arjun Humane, Prateek Roger, Alexis Williams, Andrew R. Nevmyvaka, Yuriy Rish, Irina |
| contents | Recent works have demonstrated the effectiveness of adapting pre-trained language models (LMs) for forecasting time series in the low-data regime. We build upon these findings by analyzing the effective transfer from language models to time series forecasting under various design choices including upstream post-training, time series tokenizer and language backbone size. In the low-data regime, these design choices have a significant impact on the validation loss, with clear-cut choices that outperform others. Contrary to Hernandez et al. (2021), we observe that the validation loss of the LMs continues to smoothly decrease long after the validation loss of the randomly initialized models has converged, leading to a non-vanishing transfer gap that holds across design choices. These findings not only help shed light on the effective use of compute-efficient training for time series, but also open the way for the study of modality-agnostic properties of data distributions leveraged by these models. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_21570 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Random Initialization Can't Catch Up: The Advantage of Language Model Transfer for Time Series Forecasting Riachi, Roland Rasul, Kashif Ashok, Arjun Humane, Prateek Roger, Alexis Williams, Andrew R. Nevmyvaka, Yuriy Rish, Irina Computation and Language Artificial Intelligence Machine Learning Recent works have demonstrated the effectiveness of adapting pre-trained language models (LMs) for forecasting time series in the low-data regime. We build upon these findings by analyzing the effective transfer from language models to time series forecasting under various design choices including upstream post-training, time series tokenizer and language backbone size. In the low-data regime, these design choices have a significant impact on the validation loss, with clear-cut choices that outperform others. Contrary to Hernandez et al. (2021), we observe that the validation loss of the LMs continues to smoothly decrease long after the validation loss of the randomly initialized models has converged, leading to a non-vanishing transfer gap that holds across design choices. These findings not only help shed light on the effective use of compute-efficient training for time series, but also open the way for the study of modality-agnostic properties of data distributions leveraged by these models. |
| title | Random Initialization Can't Catch Up: The Advantage of Language Model Transfer for Time Series Forecasting |
| topic | Computation and Language Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2506.21570 |