Sequence Complementor: Complementing Transformers For Time Series Forecasting with Learnable Sequences
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913636872617984 |
|---|---|
| author | Chen, Xiwen Qiu, Peijie Zhu, Wenhui Li, Huayu Wang, Hao Sotiras, Aristeidis Wang, Yalin Razi, Abolfazl |
| author_facet | Chen, Xiwen Qiu, Peijie Zhu, Wenhui Li, Huayu Wang, Hao Sotiras, Aristeidis Wang, Yalin Razi, Abolfazl |
| contents | Since its introduction, the transformer has shifted the development trajectory away from traditional models (e.g., RNN, MLP) in time series forecasting, which is attributed to its ability to capture global dependencies within temporal tokens. Follow-up studies have largely involved altering the tokenization and self-attention modules to better adapt Transformers for addressing special challenges like non-stationarity, channel-wise dependency, and variable correlation in time series. However, we found that the expressive capability of sequence representation is a key factor influencing Transformer performance in time forecasting after investigating several representative methods, where there is an almost linear relationship between sequence representation entropy and mean square error, with more diverse representations performing better. In this paper, we propose a novel attention mechanism with Sequence Complementors and prove feasible from an information theory perspective, where these learnable sequences are able to provide complementary information beyond current input to feed attention. We further enhance the Sequence Complementors via a diversification loss that is theoretically covered. The empirical evaluation of both long-term and short-term forecasting has confirmed its superiority over the recent state-of-the-art methods. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2501_02735 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Sequence Complementor: Complementing Transformers For Time Series Forecasting with Learnable Sequences Chen, Xiwen Qiu, Peijie Zhu, Wenhui Li, Huayu Wang, Hao Sotiras, Aristeidis Wang, Yalin Razi, Abolfazl Machine Learning Since its introduction, the transformer has shifted the development trajectory away from traditional models (e.g., RNN, MLP) in time series forecasting, which is attributed to its ability to capture global dependencies within temporal tokens. Follow-up studies have largely involved altering the tokenization and self-attention modules to better adapt Transformers for addressing special challenges like non-stationarity, channel-wise dependency, and variable correlation in time series. However, we found that the expressive capability of sequence representation is a key factor influencing Transformer performance in time forecasting after investigating several representative methods, where there is an almost linear relationship between sequence representation entropy and mean square error, with more diverse representations performing better. In this paper, we propose a novel attention mechanism with Sequence Complementors and prove feasible from an information theory perspective, where these learnable sequences are able to provide complementary information beyond current input to feed attention. We further enhance the Sequence Complementors via a diversification loss that is theoretically covered. The empirical evaluation of both long-term and short-term forecasting has confirmed its superiority over the recent state-of-the-art methods. |
| title | Sequence Complementor: Complementing Transformers For Time Series Forecasting with Learnable Sequences |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2501.02735 |