Understanding Transformers for Time Series: Rank Structure, Flow-of-ranks, and Compressibility
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866908575132024832 |
|---|---|
| author | Yu, Annan Maddix, Danielle C. Han, Boran Zhang, Xiyuan Ansari, Abdul Fatir Shchur, Oleksandr Faloutsos, Christos Wilson, Andrew Gordon Mahoney, Michael W. Wang, Yuyang |
| author_facet | Yu, Annan Maddix, Danielle C. Han, Boran Zhang, Xiyuan Ansari, Abdul Fatir Shchur, Oleksandr Faloutsos, Christos Wilson, Andrew Gordon Mahoney, Michael W. Wang, Yuyang |
| contents | Transformers are widely used across data modalities, and yet the principles distilled from text models often transfer imperfectly to models trained to other modalities. In this paper, we analyze Transformers through the lens of rank structure. Our focus is on the time series setting, where the structural properties of the data differ remarkably from those of text or vision. We show that time-series embeddings, unlike text or vision, exhibit sharply decaying singular value spectra: small patch sizes and smooth continuous mappings concentrate the data into low-rank subspaces. From this, we prove that the associated $Q/K/V$ projections admit accurate low-rank approximations, and that attention layers become compressible in proportion to the decay of the embedding spectrum. We introduce the concept of flow-of-ranks, a phenomenon by which nonlinear mixing across depth inflates the rank, explaining why early layers are most amenable to compression and why ranks grow with depth. Guided by these theoretical and empirical results, we use these insights to compress Chronos, a large time series foundation model, achieving a reduction of $65\%$ in inference time and $81\%$ in memory, without loss of accuracy. Our findings provide principled guidance for allocating width, depth, and heads in time series foundation models, and for exploiting their inherent compressibility. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_03358 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Understanding Transformers for Time Series: Rank Structure, Flow-of-ranks, and Compressibility Yu, Annan Maddix, Danielle C. Han, Boran Zhang, Xiyuan Ansari, Abdul Fatir Shchur, Oleksandr Faloutsos, Christos Wilson, Andrew Gordon Mahoney, Michael W. Wang, Yuyang Machine Learning Artificial Intelligence Transformers are widely used across data modalities, and yet the principles distilled from text models often transfer imperfectly to models trained to other modalities. In this paper, we analyze Transformers through the lens of rank structure. Our focus is on the time series setting, where the structural properties of the data differ remarkably from those of text or vision. We show that time-series embeddings, unlike text or vision, exhibit sharply decaying singular value spectra: small patch sizes and smooth continuous mappings concentrate the data into low-rank subspaces. From this, we prove that the associated $Q/K/V$ projections admit accurate low-rank approximations, and that attention layers become compressible in proportion to the decay of the embedding spectrum. We introduce the concept of flow-of-ranks, a phenomenon by which nonlinear mixing across depth inflates the rank, explaining why early layers are most amenable to compression and why ranks grow with depth. Guided by these theoretical and empirical results, we use these insights to compress Chronos, a large time series foundation model, achieving a reduction of $65\%$ in inference time and $81\%$ in memory, without loss of accuracy. Our findings provide principled guidance for allocating width, depth, and heads in time series foundation models, and for exploiting their inherent compressibility. |
| title | Understanding Transformers for Time Series: Rank Structure, Flow-of-ranks, and Compressibility |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2510.03358 |