How more data can hurt: Instability and regularization in next-generation reservoir computing
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866916827566702592 |
|---|---|
| author | Zhang, Yuanzhao Santos, Edmilson Roque dos Zhang, Huixin Cornelius, Sean P. |
| author_facet | Zhang, Yuanzhao Santos, Edmilson Roque dos Zhang, Huixin Cornelius, Sean P. |
| contents | It has been found recently that more data can, counter-intuitively, hurt the performance of deep neural networks. Here, we show that a more extreme version of the phenomenon occurs in data-driven models of dynamical systems. To elucidate the underlying mechanism, we focus on next-generation reservoir computing (NGRC) -- a popular framework for learning dynamics from data. We find that, despite learning a better representation of the flow map with more training data, NGRC can adopt an ill-conditioned ``integrator'' and lose stability. We link this data-induced instability to the auxiliary dimensions created by the delayed states in NGRC. Based on these findings, we propose simple strategies to mitigate the instability, either by increasing regularization strength in tandem with data size, or by carefully introducing noise during training. Our results highlight the importance of proper regularization in data-driven modeling of dynamical systems. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2407_08641 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | How more data can hurt: Instability and regularization in next-generation reservoir computing Zhang, Yuanzhao Santos, Edmilson Roque dos Zhang, Huixin Cornelius, Sean P. Machine Learning Neural and Evolutionary Computing Dynamical Systems Adaptation and Self-Organizing Systems It has been found recently that more data can, counter-intuitively, hurt the performance of deep neural networks. Here, we show that a more extreme version of the phenomenon occurs in data-driven models of dynamical systems. To elucidate the underlying mechanism, we focus on next-generation reservoir computing (NGRC) -- a popular framework for learning dynamics from data. We find that, despite learning a better representation of the flow map with more training data, NGRC can adopt an ill-conditioned ``integrator'' and lose stability. We link this data-induced instability to the auxiliary dimensions created by the delayed states in NGRC. Based on these findings, we propose simple strategies to mitigate the instability, either by increasing regularization strength in tandem with data size, or by carefully introducing noise during training. Our results highlight the importance of proper regularization in data-driven modeling of dynamical systems. |
| title | How more data can hurt: Instability and regularization in next-generation reservoir computing |
| topic | Machine Learning Neural and Evolutionary Computing Dynamical Systems Adaptation and Self-Organizing Systems |
| url | https://arxiv.org/abs/2407.08641 |