On Dissipativity of Cross-Entropy Loss in Training ResNets
Fuente:
arXiv
Saved in:
| Main Authors: | , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916265104244736 |
|---|---|
| author | Püttschneider, Jens Faulwasser, Timm |
| author_facet | Püttschneider, Jens Faulwasser, Timm |
| contents | The training of ResNets and neural ODEs can be formulated and analyzed from the perspective of optimal control. This paper proposes a dissipative formulation of the training of ResNets and neural ODEs for classification problems by including a variant of the cross-entropy as a regularization in the stage cost. Based on the dissipative formulation of the training, we prove that the trained ResNet exhibit the turnpike phenomenon. We then illustrate that the training exhibits the turnpike phenomenon by training on the two spirals and MNIST datasets. This can be used to find very shallow networks suitable for a given classification task. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_19013 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | On Dissipativity of Cross-Entropy Loss in Training ResNets Püttschneider, Jens Faulwasser, Timm Machine Learning Systems and Control Optimization and Control The training of ResNets and neural ODEs can be formulated and analyzed from the perspective of optimal control. This paper proposes a dissipative formulation of the training of ResNets and neural ODEs for classification problems by including a variant of the cross-entropy as a regularization in the stage cost. Based on the dissipative formulation of the training, we prove that the trained ResNet exhibit the turnpike phenomenon. We then illustrate that the training exhibits the turnpike phenomenon by training on the two spirals and MNIST datasets. This can be used to find very shallow networks suitable for a given classification task. |
| title | On Dissipativity of Cross-Entropy Loss in Training ResNets |
| topic | Machine Learning Systems and Control Optimization and Control |
| url | https://arxiv.org/abs/2405.19013 |