Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2023
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2312.11166 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917836772868096 |
|---|---|
| author | Brantner, Benedikt de Romemont, Guillaume Kraus, Michael Li, Zeyuan |
| author_facet | Brantner, Benedikt de Romemont, Guillaume Kraus, Michael Li, Zeyuan |
| contents | Two of the many trends in neural network research of the past few years have been (i) the learning of dynamical systems, especially with recurrent neural networks such as long short-term memory networks (LSTMs) and (ii) the introduction of transformer neural networks for natural language processing (NLP) tasks.
While some work has been performed on the intersection of these two trends, those efforts were largely limited to using the vanilla transformer directly without adjusting its architecture for the setting of a physical system.
In this work we develop a transformer-inspired neural network and use it to learn a dynamical system. We (for the first time) change the activation function of the attention layer to imbue the transformer with structure-preserving properties to improve long-term stability. This is shown to be of great advantage when applying the neural network to learning the trajectory of a rigid body. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2312_11166 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | Volume-Preserving Transformers for Learning Time Series Data with Structure Brantner, Benedikt de Romemont, Guillaume Kraus, Michael Li, Zeyuan Numerical Analysis Machine Learning 68T07, 65D30, 37M15, 65P10 Two of the many trends in neural network research of the past few years have been (i) the learning of dynamical systems, especially with recurrent neural networks such as long short-term memory networks (LSTMs) and (ii) the introduction of transformer neural networks for natural language processing (NLP) tasks. While some work has been performed on the intersection of these two trends, those efforts were largely limited to using the vanilla transformer directly without adjusting its architecture for the setting of a physical system. In this work we develop a transformer-inspired neural network and use it to learn a dynamical system. We (for the first time) change the activation function of the attention layer to imbue the transformer with structure-preserving properties to improve long-term stability. This is shown to be of great advantage when applying the neural network to learning the trajectory of a rigid body. |
| title | Volume-Preserving Transformers for Learning Time Series Data with Structure |
| topic | Numerical Analysis Machine Learning 68T07, 65D30, 37M15, 65P10 |
| url | https://arxiv.org/abs/2312.11166 |