Deep Network Trainability via Persistent Subspace Orthogonality
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866912895465422848 |
|---|---|
| author | Massucco, Alex Murari, Davide Schönlieb, Carola-Bibiane |
| author_facet | Massucco, Alex Murari, Davide Schönlieb, Carola-Bibiane |
| contents | Training neural networks via backpropagation is often hindered by vanishing or exploding gradients. In this work, we design architectures that mitigate these issues by analyzing and controlling the network Jacobian. We first provide a unified characterization for a class of networks with orthogonal Jacobian including known architectures and yielding new trainable designs. We then introduce the relaxed notion of persistent subspace orthogonality. This applies to a broader class of networks whose Jacobians are isometries only on a non-trivial subspace. We propose practical mechanisms to enforce this condition and empirically show that it is necessary to sufficiently preserve the gradient norms during backpropagation, enabling the training of very deep networks. We support our theory with extensive experiments. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_02882 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Deep Network Trainability via Persistent Subspace Orthogonality Massucco, Alex Murari, Davide Schönlieb, Carola-Bibiane Machine Learning Numerical Analysis Training neural networks via backpropagation is often hindered by vanishing or exploding gradients. In this work, we design architectures that mitigate these issues by analyzing and controlling the network Jacobian. We first provide a unified characterization for a class of networks with orthogonal Jacobian including known architectures and yielding new trainable designs. We then introduce the relaxed notion of persistent subspace orthogonality. This applies to a broader class of networks whose Jacobians are isometries only on a non-trivial subspace. We propose practical mechanisms to enforce this condition and empirically show that it is necessary to sufficiently preserve the gradient norms during backpropagation, enabling the training of very deep networks. We support our theory with extensive experiments. |
| title | Deep Network Trainability via Persistent Subspace Orthogonality |
| topic | Machine Learning Numerical Analysis |
| url | https://arxiv.org/abs/2508.02882 |