Spectral complexity of deep neural networks
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913846005858304 |
|---|---|
| author | Di Lillo, Simmaco Marinucci, Domenico Salvi, Michele Vigogna, Stefano |
| author_facet | Di Lillo, Simmaco Marinucci, Domenico Salvi, Michele Vigogna, Stefano |
| contents | It is well-known that randomly initialized, push-forward, fully-connected neural networks weakly converge to isotropic Gaussian processes, in the limit where the width of all layers goes to infinity. In this paper, we propose to use the angular power spectrum of the limiting field to characterize the complexity of the network architecture. In particular, we define sequences of random variables associated with the angular power spectrum, and provide a full characterization of the network complexity in terms of the asymptotic distribution of these sequences as the depth diverges. On this basis, we classify neural networks as low-disorder, sparse, or high-disorder; we show how this classification highlights a number of distinct features for standard activation functions, and in particular, sparsity properties of ReLU networks. Our theoretical results are also validated by numerical simulations. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_09541 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Spectral complexity of deep neural networks Di Lillo, Simmaco Marinucci, Domenico Salvi, Michele Vigogna, Stefano Machine Learning Probability 68T07, 60G60, 33C55, 62M15 It is well-known that randomly initialized, push-forward, fully-connected neural networks weakly converge to isotropic Gaussian processes, in the limit where the width of all layers goes to infinity. In this paper, we propose to use the angular power spectrum of the limiting field to characterize the complexity of the network architecture. In particular, we define sequences of random variables associated with the angular power spectrum, and provide a full characterization of the network complexity in terms of the asymptotic distribution of these sequences as the depth diverges. On this basis, we classify neural networks as low-disorder, sparse, or high-disorder; we show how this classification highlights a number of distinct features for standard activation functions, and in particular, sparsity properties of ReLU networks. Our theoretical results are also validated by numerical simulations. |
| title | Spectral complexity of deep neural networks |
| topic | Machine Learning Probability 68T07, 60G60, 33C55, 62M15 |
| url | https://arxiv.org/abs/2405.09541 |