Saved in:
| Main Authors: | , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2408.01517 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910005669658624 |
|---|---|
| author | Chen, Thomas Ewald, Patrícia Muñoz |
| author_facet | Chen, Thomas Ewald, Patrícia Muñoz |
| contents | We prove that the standard gradient flow in parameter space that underlies many training algorithms in deep learning can be continuously deformed into an adapted gradient flow which yields (constrained) Euclidean gradient flow in output space. Moreover, for the $L^{2}$ loss, if the Jacobian of the outputs with respect to the parameters is full rank (for fixed training data), then the time variable can be reparametrized so that the resulting flow is simply linear interpolation, and a global minimum can be achieved. For the cross-entropy loss, under the same rank condition and assuming the labels have positive components, we derive an explicit formula for the unique global minimum. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_01517 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Gradient flow in parameter space is equivalent to linear interpolation in output space Chen, Thomas Ewald, Patrícia Muñoz Machine Learning Artificial Intelligence Mathematical Physics Optimization and Control 62M45, 37C10 We prove that the standard gradient flow in parameter space that underlies many training algorithms in deep learning can be continuously deformed into an adapted gradient flow which yields (constrained) Euclidean gradient flow in output space. Moreover, for the $L^{2}$ loss, if the Jacobian of the outputs with respect to the parameters is full rank (for fixed training data), then the time variable can be reparametrized so that the resulting flow is simply linear interpolation, and a global minimum can be achieved. For the cross-entropy loss, under the same rank condition and assuming the labels have positive components, we derive an explicit formula for the unique global minimum. |
| title | Gradient flow in parameter space is equivalent to linear interpolation in output space |
| topic | Machine Learning Artificial Intelligence Mathematical Physics Optimization and Control 62M45, 37C10 |
| url | https://arxiv.org/abs/2408.01517 |