Entropic Regularization in the Deep Linear Network
Fuente:
arXiv
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912752408199168 |
|---|---|
| author | Chen, Alan Kotwal, Tejas Menon, Govind |
| author_facet | Chen, Alan Kotwal, Tejas Menon, Govind |
| contents | We study regularization for the deep linear network (DLN) using the entropy formula introduced in arXiv:2509.09088. The equilibria and gradient flow of the free energy on the Riemannian manifold of end-to-end maps of the DLN are characterized for energies that depend symmetrically on the singular values of the end-to-end matrix.
The only equilibria are minimizers and the set of minimizers is an orbit of the orthogonal group. In contrast with random matrix theory there is no singular value repulsion. The corresponding gradient flow reduces to a one-dimensional ordinary differential equation whose solution gives explicit relaxation rates toward the minimizers. We also study the concavity of the entropy in the chamber of singular values. The entropy is shown to be strictly concave in the Euclidean geometry on the chamber but not in the Riemannian geometry defined by the DLN metric. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_06137 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Entropic Regularization in the Deep Linear Network Chen, Alan Kotwal, Tejas Menon, Govind Neural and Evolutionary Computing Dynamical Systems Probability 37N40, 53C20, 15A18, 68T07 We study regularization for the deep linear network (DLN) using the entropy formula introduced in arXiv:2509.09088. The equilibria and gradient flow of the free energy on the Riemannian manifold of end-to-end maps of the DLN are characterized for energies that depend symmetrically on the singular values of the end-to-end matrix. The only equilibria are minimizers and the set of minimizers is an orbit of the orthogonal group. In contrast with random matrix theory there is no singular value repulsion. The corresponding gradient flow reduces to a one-dimensional ordinary differential equation whose solution gives explicit relaxation rates toward the minimizers. We also study the concavity of the entropy in the chamber of singular values. The entropy is shown to be strictly concave in the Euclidean geometry on the chamber but not in the Riemannian geometry defined by the DLN metric. |
| title | Entropic Regularization in the Deep Linear Network |
| topic | Neural and Evolutionary Computing Dynamical Systems Probability 37N40, 53C20, 15A18, 68T07 |
| url | https://arxiv.org/abs/2512.06137 |