Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2508.03222 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914259668041728 |
|---|---|
| author | D'Inverno, Giuseppe Alessio Hu, Zhiyuan Davy, Leo Unser, Michael Rozza, Gianluigi Dong, Jonathan |
| author_facet | D'Inverno, Giuseppe Alessio Hu, Zhiyuan Davy, Leo Unser, Michael Rozza, Gianluigi Dong, Jonathan |
| contents | Information propagation characterizes how input correlations evolve across layers in deep neural networks. This framework has been well studied using mean-field theory, which assumes infinitely wide networks. However, these assumptions break down for practical, finite-size networks. In this work, we study information propagation in randomly initialized neural networks with finite width and reveal that the boundary between ordered and chaotic regimes exhibits a fractal structure. This shows the fundamental complexity of neural network dynamics, in a setting that is independent of input data and optimization. To extend this analysis beyond multilayer perceptrons, we leverage recently introduced Fourier-based structured transforms, and show that information propagation in convolutional neural networks also follow the same behavior. In practice, our investigation highlights the importance of finite network depth with respect to the tradeoff between separation and robustness. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_03222 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Revisiting Deep Information Propagation: Fractal Frontier and Finite-size Effects D'Inverno, Giuseppe Alessio Hu, Zhiyuan Davy, Leo Unser, Michael Rozza, Gianluigi Dong, Jonathan Machine Learning Information propagation characterizes how input correlations evolve across layers in deep neural networks. This framework has been well studied using mean-field theory, which assumes infinitely wide networks. However, these assumptions break down for practical, finite-size networks. In this work, we study information propagation in randomly initialized neural networks with finite width and reveal that the boundary between ordered and chaotic regimes exhibits a fractal structure. This shows the fundamental complexity of neural network dynamics, in a setting that is independent of input data and optimization. To extend this analysis beyond multilayer perceptrons, we leverage recently introduced Fourier-based structured transforms, and show that information propagation in convolutional neural networks also follow the same behavior. In practice, our investigation highlights the importance of finite network depth with respect to the tradeoff between separation and robustness. |
| title | Revisiting Deep Information Propagation: Fractal Frontier and Finite-size Effects |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2508.03222 |