Towards an Optimal Control Perspective of ResNet Training
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911024661135360 |
|---|---|
| author | Püttschneider, Jens Heilig, Simon Fischer, Asja Faulwasser, Timm |
| author_facet | Püttschneider, Jens Heilig, Simon Fischer, Asja Faulwasser, Timm |
| contents | We propose a training formulation for ResNets reflecting an optimal control problem that is applicable for standard architectures and general loss functions. We suggest bridging both worlds via penalizing intermediate outputs of hidden states corresponding to stage cost terms in optimal control. For standard ResNets, we obtain intermediate outputs by propagating the state through the subsequent skip connections and the output layer. We demonstrate that our training dynamic biases the weights of the unnecessary deeper residual layers to vanish. This indicates the potential for a theory-grounded layer pruning strategy. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_21453 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Towards an Optimal Control Perspective of ResNet Training Püttschneider, Jens Heilig, Simon Fischer, Asja Faulwasser, Timm Machine Learning Systems and Control Optimization and Control We propose a training formulation for ResNets reflecting an optimal control problem that is applicable for standard architectures and general loss functions. We suggest bridging both worlds via penalizing intermediate outputs of hidden states corresponding to stage cost terms in optimal control. For standard ResNets, we obtain intermediate outputs by propagating the state through the subsequent skip connections and the output layer. We demonstrate that our training dynamic biases the weights of the unnecessary deeper residual layers to vanish. This indicates the potential for a theory-grounded layer pruning strategy. |
| title | Towards an Optimal Control Perspective of ResNet Training |
| topic | Machine Learning Systems and Control Optimization and Control |
| url | https://arxiv.org/abs/2506.21453 |