A Survey of Dimension Estimation Methods
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866909694863343616 |
|---|---|
| author | Binnie, James A. D. Dłotko, Paweł Harvey, John Malinowski, Jakub Yim, Ka Man |
| author_facet | Binnie, James A. D. Dłotko, Paweł Harvey, John Malinowski, Jakub Yim, Ka Man |
| contents | It is a standard assumption that datasets in high dimension have an internal structure which means that they in fact lie on, or near, subsets of a lower dimension. In many instances it is important to understand the real dimension of the data, hence the complexity of the dataset at hand. A great variety of dimension estimators have been developed to find the intrinsic dimension of the data but there is little guidance on how to reliably use these estimators.
This survey reviews a wide range of dimension estimation methods, categorising them by the geometric information they exploit: tangential estimators which detect a local affine structure; parametric estimators which rely on dimension-dependent probability distributions; and estimators which use topological or metric invariants.
The paper evaluates the performance of these methods, as well as investigating varying responses to curvature and noise. Key issues addressed include robustness to hyperparameter selection, sample size requirements, accuracy in high dimensions, precision, and performance on non-linear geometries. In identifying the best hyperparameters for benchmark datasets, overfitting is frequent, indicating that many estimators may not generalise well beyond the datasets on which they have been tested. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2507_13887 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | A Survey of Dimension Estimation Methods Binnie, James A. D. Dłotko, Paweł Harvey, John Malinowski, Jakub Yim, Ka Man Machine Learning Differential Geometry Metric Geometry Statistics Theory 62R40 (Primary) 62R30, 62R07, 62G05, 53Z50 (Secondary) It is a standard assumption that datasets in high dimension have an internal structure which means that they in fact lie on, or near, subsets of a lower dimension. In many instances it is important to understand the real dimension of the data, hence the complexity of the dataset at hand. A great variety of dimension estimators have been developed to find the intrinsic dimension of the data but there is little guidance on how to reliably use these estimators. This survey reviews a wide range of dimension estimation methods, categorising them by the geometric information they exploit: tangential estimators which detect a local affine structure; parametric estimators which rely on dimension-dependent probability distributions; and estimators which use topological or metric invariants. The paper evaluates the performance of these methods, as well as investigating varying responses to curvature and noise. Key issues addressed include robustness to hyperparameter selection, sample size requirements, accuracy in high dimensions, precision, and performance on non-linear geometries. In identifying the best hyperparameters for benchmark datasets, overfitting is frequent, indicating that many estimators may not generalise well beyond the datasets on which they have been tested. |
| title | A Survey of Dimension Estimation Methods |
| topic | Machine Learning Differential Geometry Metric Geometry Statistics Theory 62R40 (Primary) 62R30, 62R07, 62G05, 53Z50 (Secondary) |
| url | https://arxiv.org/abs/2507.13887 |