Approximating Simple ReLU Networks based on Spectral Decomposition of Fisher Information
Fuente:
arXiv
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911674631454720 |
|---|---|
| author | Ho, Ka Long Keith Takeishi, Yoshinari Takeuchi, Junichi |
| author_facet | Ho, Ka Long Keith Takeishi, Yoshinari Takeuchi, Junichi |
| contents | Properties of Fisher information matrices of 2-layer neural ReLU networks with random hidden weights are studied. For these networks, it is known that the eigenvalue distribution highly concentrates on several eigenspaces approximately. In particular, the eigenvalues for the first three eigenspaces account for 97.7% of the trace of the Fisher information matrix, independently of the number of parameters. In this paper, we identify the function spaces which correspond to those major eigenspaces. This function space consists of the spherical harmonic functions whose orders are not greater than 2. This result relates to the Mercer decomposition of the neural tangent kernels. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_17907 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Approximating Simple ReLU Networks based on Spectral Decomposition of Fisher Information Ho, Ka Long Keith Takeishi, Yoshinari Takeuchi, Junichi Machine Learning Properties of Fisher information matrices of 2-layer neural ReLU networks with random hidden weights are studied. For these networks, it is known that the eigenvalue distribution highly concentrates on several eigenspaces approximately. In particular, the eigenvalues for the first three eigenspaces account for 97.7% of the trace of the Fisher information matrix, independently of the number of parameters. In this paper, we identify the function spaces which correspond to those major eigenspaces. This function space consists of the spherical harmonic functions whose orders are not greater than 2. This result relates to the Mercer decomposition of the neural tangent kernels. |
| title | Approximating Simple ReLU Networks based on Spectral Decomposition of Fisher Information |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2505.17907 |