Vanilla Feedforward Neural Networks as a Discretization of Dynamical Systems
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2022
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866910669595475968 |
|---|---|
| author | Duan, Yifei Li, Li'ang Ji, Guanghua Cai, Yongqiang |
| author_facet | Duan, Yifei Li, Li'ang Ji, Guanghua Cai, Yongqiang |
| contents | Deep learning has made significant applications in the field of data science and natural science. Some studies have linked deep neural networks to dynamic systems, but the network structure is restricted to the residual network. It is known that residual networks can be regarded as a numerical discretization of dynamic systems. In this paper, we back to the classical network structure and prove that the vanilla feedforward networks could also be a numerical discretization of dynamic systems, where the width of the network is equal to the dimension of the input and output. Our proof is based on the properties of the leaky-ReLU function and the numerical technique of splitting method to solve differential equations. Our results could provide a new perspective for understanding the approximation properties of feedforward neural networks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2209_10909 |
| institution | arXiv |
| publishDate | 2022 |
| record_format | arxiv |
| spellingShingle | Vanilla Feedforward Neural Networks as a Discretization of Dynamical Systems Duan, Yifei Li, Li'ang Ji, Guanghua Cai, Yongqiang Machine Learning Numerical Analysis 68T07, 65P99, 65Z05, 41A65 Deep learning has made significant applications in the field of data science and natural science. Some studies have linked deep neural networks to dynamic systems, but the network structure is restricted to the residual network. It is known that residual networks can be regarded as a numerical discretization of dynamic systems. In this paper, we back to the classical network structure and prove that the vanilla feedforward networks could also be a numerical discretization of dynamic systems, where the width of the network is equal to the dimension of the input and output. Our proof is based on the properties of the leaky-ReLU function and the numerical technique of splitting method to solve differential equations. Our results could provide a new perspective for understanding the approximation properties of feedforward neural networks. |
| title | Vanilla Feedforward Neural Networks as a Discretization of Dynamical Systems |
| topic | Machine Learning Numerical Analysis 68T07, 65P99, 65Z05, 41A65 |
| url | https://arxiv.org/abs/2209.10909 |