LLM Pretraining Shapes a Generalizable Manifold: Insights into Cross-Modal Transfer to Time Series
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2026
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866911699953516544 |
|---|---|
| author | Roger, Alexis Humane, Prateek Tai, Zhenghan Legate, Gwen Mircea, Andrei Feofanov, Vasilii Rish, Irina |
| author_facet | Roger, Alexis Humane, Prateek Tai, Zhenghan Legate, Gwen Mircea, Andrei Feofanov, Vasilii Rish, Irina |
| contents | Can language-pretrained transformers become effective time-series forecasters, and why? In this paper, we show that cross-modal transfer arises because language pretraining preconditions time series training with a reusable manifold. A linear probe on frozen LLM states decodes realistic time-series trajectories without paired supervision, and retrieval in this projected space yields competitive forecasts, showing that structure and dynamics exist before finetuning. Pretrained initialization also improves optimization, producing coherent gradients and a highly anisotropic loss landscape unlike random initialization. Finetuning then acts as low-dimensional alignment, reusing existing directions rather than learning temporal primitives from scratch, as evidenced by low-rank updates, subspace alignment, and shared features for periodicity, trend, and repetition. Together, these results support a geometric account of LLM-to-time-series transfer: language pretraining builds the manifold, and finetuning projects numerical dynamics onto task-relevant directions. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2605_20449 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | LLM Pretraining Shapes a Generalizable Manifold: Insights into Cross-Modal Transfer to Time Series Roger, Alexis Humane, Prateek Tai, Zhenghan Legate, Gwen Mircea, Andrei Feofanov, Vasilii Rish, Irina Machine Learning Artificial Intelligence Can language-pretrained transformers become effective time-series forecasters, and why? In this paper, we show that cross-modal transfer arises because language pretraining preconditions time series training with a reusable manifold. A linear probe on frozen LLM states decodes realistic time-series trajectories without paired supervision, and retrieval in this projected space yields competitive forecasts, showing that structure and dynamics exist before finetuning. Pretrained initialization also improves optimization, producing coherent gradients and a highly anisotropic loss landscape unlike random initialization. Finetuning then acts as low-dimensional alignment, reusing existing directions rather than learning temporal primitives from scratch, as evidenced by low-rank updates, subspace alignment, and shared features for periodicity, trend, and repetition. Together, these results support a geometric account of LLM-to-time-series transfer: language pretraining builds the manifold, and finetuning projects numerical dynamics onto task-relevant directions. |
| title | LLM Pretraining Shapes a Generalizable Manifold: Insights into Cross-Modal Transfer to Time Series |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2605.20449 |