From NeurODEs to AutoencODEs: a mean-field control framework for width-varying Neural Networks
Fuente:
arXiv
Salvato in:
| Autori principali: | , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2023
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866916655554101248 |
|---|---|
| author | Cipriani, Cristina Fornasier, Massimo Scagliotti, Alessandro |
| author_facet | Cipriani, Cristina Fornasier, Massimo Scagliotti, Alessandro |
| contents | The connection between Residual Neural Networks (ResNets) and continuous-time control systems (known as NeurODEs) has led to a mathematical analysis of neural networks which has provided interesting results of both theoretical and practical significance. However, by construction, NeurODEs have been limited to describing constant-width layers, making them unsuitable for modeling deep learning architectures with layers of variable width. In this paper, we propose a continuous-time Autoencoder, which we call AutoencODE, based on a modification of the controlled field that drives the dynamics. This adaptation enables the extension of the mean-field control framework originally devised for conventional NeurODEs. In this setting, we tackle the case of low Tikhonov regularization, resulting in potentially non-convex cost landscapes. While the global results obtained for high Tikhonov regularization may not hold globally, we show that many of them can be recovered in regions where the loss function is locally convex. Inspired by our theoretical findings, we develop a training method tailored to this specific type of Autoencoders with residual connections, and we validate our approach through numerical experiments conducted on various examples. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2307_02279 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | From NeurODEs to AutoencODEs: a mean-field control framework for width-varying Neural Networks Cipriani, Cristina Fornasier, Massimo Scagliotti, Alessandro Optimization and Control Machine Learning Systems and Control The connection between Residual Neural Networks (ResNets) and continuous-time control systems (known as NeurODEs) has led to a mathematical analysis of neural networks which has provided interesting results of both theoretical and practical significance. However, by construction, NeurODEs have been limited to describing constant-width layers, making them unsuitable for modeling deep learning architectures with layers of variable width. In this paper, we propose a continuous-time Autoencoder, which we call AutoencODE, based on a modification of the controlled field that drives the dynamics. This adaptation enables the extension of the mean-field control framework originally devised for conventional NeurODEs. In this setting, we tackle the case of low Tikhonov regularization, resulting in potentially non-convex cost landscapes. While the global results obtained for high Tikhonov regularization may not hold globally, we show that many of them can be recovered in regions where the loss function is locally convex. Inspired by our theoretical findings, we develop a training method tailored to this specific type of Autoencoders with residual connections, and we validate our approach through numerical experiments conducted on various examples. |
| title | From NeurODEs to AutoencODEs: a mean-field control framework for width-varying Neural Networks |
| topic | Optimization and Control Machine Learning Systems and Control |
| url | https://arxiv.org/abs/2307.02279 |