Some Theoretical Results on Layerwise Effective Dimension Oscillations in Finite Width ReLU Networks
Fuente:
arXiv
Enregistré dans:
| Auteur principal: | |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866911085248905216 |
|---|---|
| author | Makwana, Darshan |
| author_facet | Makwana, Darshan |
| contents | We analyze the layerwise effective dimension (rank of the feature matrix) in fully-connected ReLU networks of finite width. Specifically, for a fixed batch of $m$ inputs and random Gaussian weights, we derive closed-form expressions for the expected rank of the \$m\times n\$ hidden activation matrices. Our main result shows that $\mathbb{E}[EDim(\ell)]=m[1-(1-2/π)^\ell]+O(e^{-c m})$ so that the rank deficit decays geometrically with ratio $1-2 / π\approx 0.3634$. We also prove a sub-Gaussian concentration bound, and identify the "revival" depths at which the expected rank attains local maxima. In particular, these peaks occur at depths $\ell_k^*\approx(k+1/2)π/\log(1/ρ)$ with height $\approx (1-e^{-π/2}) m \approx 0.79m$. We further show that this oscillatory rank behavior is a finite-width phenomenon: under orthogonal weight initialization or strong negative-slope leaky-ReLU, the rank remains (nearly) full. These results provide a precise characterization of how random ReLU layers alternately collapse and partially revive the subspace of input variations, adding nuance to prior work on expressivity of deep networks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_07675 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Some Theoretical Results on Layerwise Effective Dimension Oscillations in Finite Width ReLU Networks Makwana, Darshan Machine Learning We analyze the layerwise effective dimension (rank of the feature matrix) in fully-connected ReLU networks of finite width. Specifically, for a fixed batch of $m$ inputs and random Gaussian weights, we derive closed-form expressions for the expected rank of the \$m\times n\$ hidden activation matrices. Our main result shows that $\mathbb{E}[EDim(\ell)]=m[1-(1-2/π)^\ell]+O(e^{-c m})$ so that the rank deficit decays geometrically with ratio $1-2 / π\approx 0.3634$. We also prove a sub-Gaussian concentration bound, and identify the "revival" depths at which the expected rank attains local maxima. In particular, these peaks occur at depths $\ell_k^*\approx(k+1/2)π/\log(1/ρ)$ with height $\approx (1-e^{-π/2}) m \approx 0.79m$. We further show that this oscillatory rank behavior is a finite-width phenomenon: under orthogonal weight initialization or strong negative-slope leaky-ReLU, the rank remains (nearly) full. These results provide a precise characterization of how random ReLU layers alternately collapse and partially revive the subspace of input variations, adding nuance to prior work on expressivity of deep networks. |
| title | Some Theoretical Results on Layerwise Effective Dimension Oscillations in Finite Width ReLU Networks |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2507.07675 |