On the Stability of the Jacobian Matrix in Deep Neural Networks
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866915631676260352 |
|---|---|
| author | Dadoun, Benjamin Hayou, Soufiane Salam, Hanan Seddik, Mohamed El Amine Youssef, Pierre |
| author_facet | Dadoun, Benjamin Hayou, Soufiane Salam, Hanan Seddik, Mohamed El Amine Youssef, Pierre |
| contents | Deep neural networks are known to suffer from exploding or vanishing gradients as depth increases, a phenomenon closely tied to the spectral behavior of the input-output Jacobian. Prior work has identified critical initialization schemes that ensure Jacobian stability, but these analyses are typically restricted to fully connected networks with i.i.d. weights. In this work, we go significantly beyond these limitations: we establish a general stability theorem for deep neural networks that accommodates sparsity (such as that introduced by pruning) and non-i.i.d., weakly correlated weights (e.g. induced by training). Our results rely on recent advances in random matrix theory, and provide rigorous guarantees for spectral stability in a much broader class of network models. This extends the theoretical foundation for initialization schemes in modern neural networks with structured and dependent randomness. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2506_08764 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | On the Stability of the Jacobian Matrix in Deep Neural Networks Dadoun, Benjamin Hayou, Soufiane Salam, Hanan Seddik, Mohamed El Amine Youssef, Pierre Machine Learning 68T07, 60B20 Deep neural networks are known to suffer from exploding or vanishing gradients as depth increases, a phenomenon closely tied to the spectral behavior of the input-output Jacobian. Prior work has identified critical initialization schemes that ensure Jacobian stability, but these analyses are typically restricted to fully connected networks with i.i.d. weights. In this work, we go significantly beyond these limitations: we establish a general stability theorem for deep neural networks that accommodates sparsity (such as that introduced by pruning) and non-i.i.d., weakly correlated weights (e.g. induced by training). Our results rely on recent advances in random matrix theory, and provide rigorous guarantees for spectral stability in a much broader class of network models. This extends the theoretical foundation for initialization schemes in modern neural networks with structured and dependent randomness. |
| title | On the Stability of the Jacobian Matrix in Deep Neural Networks |
| topic | Machine Learning 68T07, 60B20 |
| url | https://arxiv.org/abs/2506.08764 |