Towards Understanding Gradient Flow Dynamics of Homogeneous Neural Networks Beyond the Origin
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866916739110928384 |
|---|---|
| author | Kumar, Akshay Haupt, Jarvis |
| author_facet | Kumar, Akshay Haupt, Jarvis |
| contents | Recent works exploring the training dynamics of homogeneous neural network weights under gradient flow with small initialization have established that in the early stages of training, the weights remain small and near the origin, but converge in direction. Building on this, the current paper studies the gradient flow dynamics of homogeneous neural networks with locally Lipschitz gradients, after they escape the origin. Insights gained from this analysis are used to characterize the first saddle point encountered by gradient flow after escaping the origin. Also, it is shown that for homogeneous feed-forward neural networks, under certain conditions, the sparsity structure emerging among the weights before the escape is preserved after escaping the origin and until reaching the next saddle point. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_15952 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Towards Understanding Gradient Flow Dynamics of Homogeneous Neural Networks Beyond the Origin Kumar, Akshay Haupt, Jarvis Machine Learning Optimization and Control Recent works exploring the training dynamics of homogeneous neural network weights under gradient flow with small initialization have established that in the early stages of training, the weights remain small and near the origin, but converge in direction. Building on this, the current paper studies the gradient flow dynamics of homogeneous neural networks with locally Lipschitz gradients, after they escape the origin. Insights gained from this analysis are used to characterize the first saddle point encountered by gradient flow after escaping the origin. Also, it is shown that for homogeneous feed-forward neural networks, under certain conditions, the sparsity structure emerging among the weights before the escape is preserved after escaping the origin and until reaching the next saddle point. |
| title | Towards Understanding Gradient Flow Dynamics of Homogeneous Neural Networks Beyond the Origin |
| topic | Machine Learning Optimization and Control |
| url | https://arxiv.org/abs/2502.15952 |