On the Limitations of Fractal Dimension as a Measure of Generalization
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916464053714944 |
|---|---|
| author | Tan, Charlie B. García-Redondo, Inés Wang, Qiquan Bronstein, Michael M. Monod, Anthea |
| author_facet | Tan, Charlie B. García-Redondo, Inés Wang, Qiquan Bronstein, Michael M. Monod, Anthea |
| contents | Bounding and predicting the generalization gap of overparameterized neural networks remains a central open problem in theoretical machine learning. There is a recent and growing body of literature that proposes the framework of fractals to model optimization trajectories of neural networks, motivating generalization bounds and measures based on the fractal dimension of the trajectory. Notably, the persistent homology dimension has been proposed to correlate with the generalization gap. This paper performs an empirical evaluation of these persistent homology-based generalization measures, with an in-depth statistical analysis. Our study reveals confounding effects in the observed correlation between generalization and topological measures due to the variation of hyperparameters. We also observe that fractal dimension fails to predict generalization of models trained from poor initializations. We lastly reveal the intriguing manifestation of model-wise double descent in these topological generalization measures. Our work forms a basis for a deeper investigation of the causal relationships between fractal geometry, topological data analysis, and neural network optimization. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_02234 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | On the Limitations of Fractal Dimension as a Measure of Generalization Tan, Charlie B. García-Redondo, Inés Wang, Qiquan Bronstein, Michael M. Monod, Anthea Machine Learning Artificial Intelligence Dynamical Systems Bounding and predicting the generalization gap of overparameterized neural networks remains a central open problem in theoretical machine learning. There is a recent and growing body of literature that proposes the framework of fractals to model optimization trajectories of neural networks, motivating generalization bounds and measures based on the fractal dimension of the trajectory. Notably, the persistent homology dimension has been proposed to correlate with the generalization gap. This paper performs an empirical evaluation of these persistent homology-based generalization measures, with an in-depth statistical analysis. Our study reveals confounding effects in the observed correlation between generalization and topological measures due to the variation of hyperparameters. We also observe that fractal dimension fails to predict generalization of models trained from poor initializations. We lastly reveal the intriguing manifestation of model-wise double descent in these topological generalization measures. Our work forms a basis for a deeper investigation of the causal relationships between fractal geometry, topological data analysis, and neural network optimization. |
| title | On the Limitations of Fractal Dimension as a Measure of Generalization |
| topic | Machine Learning Artificial Intelligence Dynamical Systems |
| url | https://arxiv.org/abs/2406.02234 |