Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2305.19141 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917782396862464 |
|---|---|
| author | Nivron, Omer Parthipan, Raghul Wischik, Damon J. |
| author_facet | Nivron, Omer Parthipan, Raghul Wischik, Damon J. |
| contents | We propose the Taylorformer for random processes such as time series. Its two key components are: 1) the LocalTaylor wrapper which adapts Taylor approximations (used in dynamical systems) for use in neural network-based probabilistic models, and 2) the MHA-X attention block which makes predictions in a way inspired by how Gaussian Processes' mean predictions are linear smoothings of contextual data. Taylorformer outperforms the state-of-the-art in terms of log-likelihood on 5/6 classic Neural Process tasks such as meta-learning 1D functions, and has at least a 14\% MSE improvement on forecasting tasks, including electricity, oil temperatures and exchange rates. Taylorformer approximates a consistent stochastic process and provides uncertainty-aware predictions. Our code is provided in the supplementary material. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2305_19141 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Taylorformer: Probabilistic Modelling for Random Processes including Time Series Nivron, Omer Parthipan, Raghul Wischik, Damon J. Machine Learning We propose the Taylorformer for random processes such as time series. Its two key components are: 1) the LocalTaylor wrapper which adapts Taylor approximations (used in dynamical systems) for use in neural network-based probabilistic models, and 2) the MHA-X attention block which makes predictions in a way inspired by how Gaussian Processes' mean predictions are linear smoothings of contextual data. Taylorformer outperforms the state-of-the-art in terms of log-likelihood on 5/6 classic Neural Process tasks such as meta-learning 1D functions, and has at least a 14\% MSE improvement on forecasting tasks, including electricity, oil temperatures and exchange rates. Taylorformer approximates a consistent stochastic process and provides uncertainty-aware predictions. Our code is provided in the supplementary material. |
| title | Taylorformer: Probabilistic Modelling for Random Processes including Time Series |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2305.19141 |