Comparing Prior and Learned Time Representations in Transformer Models of Timeseries
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866929597817290752 |
|---|---|
| author | Koliou, Natalia Boura, Tatiana Konstantopoulos, Stasinos Meramveliotakis, George Kosmadakis, George |
| author_facet | Koliou, Natalia Boura, Tatiana Konstantopoulos, Stasinos Meramveliotakis, George Kosmadakis, George |
| contents | What sets timeseries analysis apart from other machine learning exercises is that time representation becomes a primary aspect of the experiment setup, as it must adequately represent the temporal relations that are relevant for the application at hand. In the work described here we study wo different variations of the Transformer architecture: one where we use the fixed time representation proposed in the literature and one where the time representation is learned from the data. Our experiments use data from predicting the energy output of solar panels, a task that exhibits known periodicities (daily and seasonal) that is straight-forward to encode in the fixed time representation. Our results indicate that even in an experiment where the phenomenon is well-understood, it is difficult to encode prior knowledge due to side-effects that are difficult to mitigate. We conclude that research work is needed to work the human into the learning loop in ways that improve the robustness and trust-worthiness of the network. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_12476 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Comparing Prior and Learned Time Representations in Transformer Models of Timeseries Koliou, Natalia Boura, Tatiana Konstantopoulos, Stasinos Meramveliotakis, George Kosmadakis, George Machine Learning Artificial Intelligence What sets timeseries analysis apart from other machine learning exercises is that time representation becomes a primary aspect of the experiment setup, as it must adequately represent the temporal relations that are relevant for the application at hand. In the work described here we study wo different variations of the Transformer architecture: one where we use the fixed time representation proposed in the literature and one where the time representation is learned from the data. Our experiments use data from predicting the energy output of solar panels, a task that exhibits known periodicities (daily and seasonal) that is straight-forward to encode in the fixed time representation. Our results indicate that even in an experiment where the phenomenon is well-understood, it is difficult to encode prior knowledge due to side-effects that are difficult to mitigate. We conclude that research work is needed to work the human into the learning loop in ways that improve the robustness and trust-worthiness of the network. |
| title | Comparing Prior and Learned Time Representations in Transformer Models of Timeseries |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2411.12476 |