State Derivative Normalization for Continuous-Time Deep Neural Networks
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866909201388797952 |
|---|---|
| author | Weigand, Jonas Beintema, Gerben I. Ulmen, Jonas Görges, Daniel Tóth, Roland Schoukens, Maarten Ruskowski, Martin |
| author_facet | Weigand, Jonas Beintema, Gerben I. Ulmen, Jonas Görges, Daniel Tóth, Roland Schoukens, Maarten Ruskowski, Martin |
| contents | The importance of proper data normalization for deep neural networks is well known. However, in continuous-time state-space model estimation, it has been observed that improper normalization of either the hidden state or hidden state derivative of the model estimate, or even of the time interval can lead to numerical and optimization challenges with deep learning based methods. This results in a reduced model quality. In this contribution, we show that these three normalization tasks are inherently coupled. Due to the existence of this coupling, we propose a solution to all three normalization challenges by introducing a normalization constant at the state derivative level. We show that the appropriate choice of the normalization constant is related to the dynamics of the to-be-identified system and we derive multiple methods of obtaining an effective normalization constant. We compare and discuss all the normalization strategies on a benchmark problem based on experimental data from a cascaded tanks system and compare our results with other methods of the identification literature. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2401_02902 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | State Derivative Normalization for Continuous-Time Deep Neural Networks Weigand, Jonas Beintema, Gerben I. Ulmen, Jonas Görges, Daniel Tóth, Roland Schoukens, Maarten Ruskowski, Martin Systems and Control Machine Learning The importance of proper data normalization for deep neural networks is well known. However, in continuous-time state-space model estimation, it has been observed that improper normalization of either the hidden state or hidden state derivative of the model estimate, or even of the time interval can lead to numerical and optimization challenges with deep learning based methods. This results in a reduced model quality. In this contribution, we show that these three normalization tasks are inherently coupled. Due to the existence of this coupling, we propose a solution to all three normalization challenges by introducing a normalization constant at the state derivative level. We show that the appropriate choice of the normalization constant is related to the dynamics of the to-be-identified system and we derive multiple methods of obtaining an effective normalization constant. We compare and discuss all the normalization strategies on a benchmark problem based on experimental data from a cascaded tanks system and compare our results with other methods of the identification literature. |
| title | State Derivative Normalization for Continuous-Time Deep Neural Networks |
| topic | Systems and Control Machine Learning |
| url | https://arxiv.org/abs/2401.02902 |