Regularization Implies balancedness in the deep linear network
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866912978183389184 |
|---|---|
| author | Lindsey, Kathryn Menon, Govind |
| author_facet | Lindsey, Kathryn Menon, Govind |
| contents | We use geometric invariant theory (GIT) to study the deep linear network (DLN). The Kempf-Ness theorem is used to establish that the $L^2$ regularizer is minimized on the balanced manifold. We introduce related balancing flows using the Riemannian geometry of fibers. The balancing flow defined by the $L^2$ regularizer is shown to converge to the balanced manifold at a uniform exponential rate. The balancing flow defined by the squared moment map is computed explicitly and shown to converge globally.
This framework allows us to decompose the training dynamics into two distinct gradient flows: a regularizing flow on fibers and a learning flow on the balanced manifold. It also provides a common mathematical framework for balancedness in deep learning and linear systems theory. We use this framework to interpret balancedness in terms of fast-slow systems, model reduction and Bayesian principles. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_01137 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Regularization Implies balancedness in the deep linear network Lindsey, Kathryn Menon, Govind Machine Learning Algebraic Geometry Dynamical Systems 14L24, 37J15, 37C10, 68T07, 93B10, 53D20, 49J15, 37N40 We use geometric invariant theory (GIT) to study the deep linear network (DLN). The Kempf-Ness theorem is used to establish that the $L^2$ regularizer is minimized on the balanced manifold. We introduce related balancing flows using the Riemannian geometry of fibers. The balancing flow defined by the $L^2$ regularizer is shown to converge to the balanced manifold at a uniform exponential rate. The balancing flow defined by the squared moment map is computed explicitly and shown to converge globally. This framework allows us to decompose the training dynamics into two distinct gradient flows: a regularizing flow on fibers and a learning flow on the balanced manifold. It also provides a common mathematical framework for balancedness in deep learning and linear systems theory. We use this framework to interpret balancedness in terms of fast-slow systems, model reduction and Bayesian principles. |
| title | Regularization Implies balancedness in the deep linear network |
| topic | Machine Learning Algebraic Geometry Dynamical Systems 14L24, 37J15, 37C10, 68T07, 93B10, 53D20, 49J15, 37N40 |
| url | https://arxiv.org/abs/2511.01137 |