Approximating Simple ReLU Networks based on Spectral Decomposition of Fisher Information

Fuente: arXiv
Guardado en:
Detalles Bibliográficos
Autores principales: Ho, Ka Long Keith, Takeishi, Yoshinari, Takeuchi, Junichi
Formato: Preprint
Publicado: 2025
Materias:
Acceso en línea:
Etiquetas: Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
_version_ 1866911674631454720
author Ho, Ka Long Keith
Takeishi, Yoshinari
Takeuchi, Junichi
author_facet Ho, Ka Long Keith
Takeishi, Yoshinari
Takeuchi, Junichi
contents Properties of Fisher information matrices of 2-layer neural ReLU networks with random hidden weights are studied. For these networks, it is known that the eigenvalue distribution highly concentrates on several eigenspaces approximately. In particular, the eigenvalues for the first three eigenspaces account for 97.7% of the trace of the Fisher information matrix, independently of the number of parameters. In this paper, we identify the function spaces which correspond to those major eigenspaces. This function space consists of the spherical harmonic functions whose orders are not greater than 2. This result relates to the Mercer decomposition of the neural tangent kernels.
format Preprint
id arxiv_https___arxiv_org_abs_2505_17907
institution arXiv
publishDate 2025
record_format arxiv
spellingShingle Approximating Simple ReLU Networks based on Spectral Decomposition of Fisher Information
Ho, Ka Long Keith
Takeishi, Yoshinari
Takeuchi, Junichi
Machine Learning
Properties of Fisher information matrices of 2-layer neural ReLU networks with random hidden weights are studied. For these networks, it is known that the eigenvalue distribution highly concentrates on several eigenspaces approximately. In particular, the eigenvalues for the first three eigenspaces account for 97.7% of the trace of the Fisher information matrix, independently of the number of parameters. In this paper, we identify the function spaces which correspond to those major eigenspaces. This function space consists of the spherical harmonic functions whose orders are not greater than 2. This result relates to the Mercer decomposition of the neural tangent kernels.
title Approximating Simple ReLU Networks based on Spectral Decomposition of Fisher Information
topic Machine Learning
url https://arxiv.org/abs/2505.17907