Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2405.13867 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915094847291392 |
|---|---|
| author | Edwards, Thomas D. P. Alvey, James Alsing, Justin Nguyen, Nam H. Wandelt, Benjamin D. |
| author_facet | Edwards, Thomas D. P. Alvey, James Alsing, Justin Nguyen, Nam H. Wandelt, Benjamin D. |
| contents | Scaling laws for large language models (LLMs) have provided useful guidance in training ever larger models for predictable performance gains. Time series forecasting shares a similar sequential structure to language, and is amenable to large-scale transformer architectures. Here we show that foundational decoder-only time series transformer models exhibit analogous scaling-behavior to LLMs, with architectural details (aspect ratio and number of heads) having a minimal effect over broad ranges. We assemble a large corpus of heterogenous time series data on which to train, and establish for the first time power-law scaling with parameter count, dataset size, and training compute, spanning five orders of magnitude. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_13867 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Scaling-laws for Large Time-series Models Edwards, Thomas D. P. Alvey, James Alsing, Justin Nguyen, Nam H. Wandelt, Benjamin D. Machine Learning Artificial Intelligence Scaling laws for large language models (LLMs) have provided useful guidance in training ever larger models for predictable performance gains. Time series forecasting shares a similar sequential structure to language, and is amenable to large-scale transformer architectures. Here we show that foundational decoder-only time series transformer models exhibit analogous scaling-behavior to LLMs, with architectural details (aspect ratio and number of heads) having a minimal effect over broad ranges. We assemble a large corpus of heterogenous time series data on which to train, and establish for the first time power-law scaling with parameter count, dataset size, and training compute, spanning five orders of magnitude. |
| title | Scaling-laws for Large Time-series Models |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2405.13867 |