Evaluating Loss Landscapes from a Topology Perspective
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866917838638284800 |
|---|---|
| author | Xie, Tiankai Geniesse, Caleb Chen, Jiaqing Yang, Yaoqing Morozov, Dmitriy Mahoney, Michael W. Maciejewski, Ross Weber, Gunther H. |
| author_facet | Xie, Tiankai Geniesse, Caleb Chen, Jiaqing Yang, Yaoqing Morozov, Dmitriy Mahoney, Michael W. Maciejewski, Ross Weber, Gunther H. |
| contents | Characterizing the loss of a neural network with respect to model parameters, i.e., the loss landscape, can provide valuable insights into properties of that model. Various methods for visualizing loss landscapes have been proposed, but less emphasis has been placed on quantifying and extracting actionable and reproducible insights from these complex representations. Inspired by powerful tools from topological data analysis (TDA) for summarizing the structure of high-dimensional data, here we characterize the underlying shape (or topology) of loss landscapes, quantifying the topology to reveal new insights about neural networks. To relate our findings to the machine learning (ML) literature, we compute simple performance metrics (e.g., accuracy, error), and we characterize the local structure of loss landscapes using Hessian-based metrics (e.g., largest eigenvalue, trace, eigenvalue spectral density). Following this approach, we study established models from image pattern recognition (e.g., ResNets) and scientific ML (e.g., physics-informed neural networks), and we show how quantifying the shape of loss landscapes can provide new insights into model performance and learning dynamics. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_09807 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Evaluating Loss Landscapes from a Topology Perspective Xie, Tiankai Geniesse, Caleb Chen, Jiaqing Yang, Yaoqing Morozov, Dmitriy Mahoney, Michael W. Maciejewski, Ross Weber, Gunther H. Machine Learning Artificial Intelligence Characterizing the loss of a neural network with respect to model parameters, i.e., the loss landscape, can provide valuable insights into properties of that model. Various methods for visualizing loss landscapes have been proposed, but less emphasis has been placed on quantifying and extracting actionable and reproducible insights from these complex representations. Inspired by powerful tools from topological data analysis (TDA) for summarizing the structure of high-dimensional data, here we characterize the underlying shape (or topology) of loss landscapes, quantifying the topology to reveal new insights about neural networks. To relate our findings to the machine learning (ML) literature, we compute simple performance metrics (e.g., accuracy, error), and we characterize the local structure of loss landscapes using Hessian-based metrics (e.g., largest eigenvalue, trace, eigenvalue spectral density). Following this approach, we study established models from image pattern recognition (e.g., ResNets) and scientific ML (e.g., physics-informed neural networks), and we show how quantifying the shape of loss landscapes can provide new insights into model performance and learning dynamics. |
| title | Evaluating Loss Landscapes from a Topology Perspective |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2411.09807 |