Exploring the Manifold of Neural Networks Using Diffusion Geometry
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866909419019698176 |
|---|---|
| author | Abel, Elliott Steindl, Andrew J. Mazioud, Selma Schueler, Ellie Ogundipe, Folu Zhang, Ellen Grinspan, Yvan Reimann, Kristof Crevasse, Peyton Bhaskar, Dhananjay Viswanath, Siddharth Zhang, Yanlei Rudner, Tim G. J. Adelstein, Ian Krishnaswamy, Smita |
| author_facet | Abel, Elliott Steindl, Andrew J. Mazioud, Selma Schueler, Ellie Ogundipe, Folu Zhang, Ellen Grinspan, Yvan Reimann, Kristof Crevasse, Peyton Bhaskar, Dhananjay Viswanath, Siddharth Zhang, Yanlei Rudner, Tim G. J. Adelstein, Ian Krishnaswamy, Smita |
| contents | Drawing motivation from the manifold hypothesis, which posits that most high-dimensional data lies on or near low-dimensional manifolds, we apply manifold learning to the space of neural networks. We learn manifolds where datapoints are neural networks by introducing a distance between the hidden layer representations of the neural networks. These distances are then fed to the non-linear dimensionality reduction algorithm PHATE to create a manifold of neural networks. We characterize this manifold using features of the representation, including class separation, hierarchical cluster structure, spectral entropy, and topological structure. Our analysis reveals that high-performing networks cluster together in the manifold, displaying consistent embedding patterns across all these features. Finally, we demonstrate the utility of this approach for guiding hyperparameter optimization and neural architecture search by sampling from the manifold. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_12626 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Exploring the Manifold of Neural Networks Using Diffusion Geometry Abel, Elliott Steindl, Andrew J. Mazioud, Selma Schueler, Ellie Ogundipe, Folu Zhang, Ellen Grinspan, Yvan Reimann, Kristof Crevasse, Peyton Bhaskar, Dhananjay Viswanath, Siddharth Zhang, Yanlei Rudner, Tim G. J. Adelstein, Ian Krishnaswamy, Smita Machine Learning Drawing motivation from the manifold hypothesis, which posits that most high-dimensional data lies on or near low-dimensional manifolds, we apply manifold learning to the space of neural networks. We learn manifolds where datapoints are neural networks by introducing a distance between the hidden layer representations of the neural networks. These distances are then fed to the non-linear dimensionality reduction algorithm PHATE to create a manifold of neural networks. We characterize this manifold using features of the representation, including class separation, hierarchical cluster structure, spectral entropy, and topological structure. Our analysis reveals that high-performing networks cluster together in the manifold, displaying consistent embedding patterns across all these features. Finally, we demonstrate the utility of this approach for guiding hyperparameter optimization and neural architecture search by sampling from the manifold. |
| title | Exploring the Manifold of Neural Networks Using Diffusion Geometry |
| topic | Machine Learning |
| url | https://arxiv.org/abs/2411.12626 |