Scale-adaptive and robust intrinsic dimension estimation via optimal neighbourhood identification
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915904519929856 |
|---|---|
| author | Di Noia, Antonio Macocco, Iuri Glielmo, Aldo Laio, Alessandro Mira, Antonietta |
| author_facet | Di Noia, Antonio Macocco, Iuri Glielmo, Aldo Laio, Alessandro Mira, Antonietta |
| contents | The Intrinsic Dimension (ID) is a key concept in unsupervised learning and feature selection, as it is a lower bound to the number of variables which are necessary to describe a system. However, in almost any real-world dataset the ID depends on the scale at which the data are analysed. Quite typically at a small scale, the ID is very large, as the data are affected by measurement errors. At large scale, the ID can also appear erroneously large, due to the curvature and the topology of the manifold containing the data. In this work, we introduce an automatic protocol to select the sweet spot, namely the correct range of scales in which the ID is meaningful and useful. This protocol is based on imposing that for distances smaller than the correct scale the density of the data is constant. In the presented framework, to estimate the density it is necessary to know the ID, therefore, this condition is imposed self-consistently. We illustrate the usefulness and robustness of this procedure to noise by benchmarks on artificial and real-world datasets. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_15132 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Scale-adaptive and robust intrinsic dimension estimation via optimal neighbourhood identification Di Noia, Antonio Macocco, Iuri Glielmo, Aldo Laio, Alessandro Mira, Antonietta Machine Learning Statistics Theory Computation Methodology The Intrinsic Dimension (ID) is a key concept in unsupervised learning and feature selection, as it is a lower bound to the number of variables which are necessary to describe a system. However, in almost any real-world dataset the ID depends on the scale at which the data are analysed. Quite typically at a small scale, the ID is very large, as the data are affected by measurement errors. At large scale, the ID can also appear erroneously large, due to the curvature and the topology of the manifold containing the data. In this work, we introduce an automatic protocol to select the sweet spot, namely the correct range of scales in which the ID is meaningful and useful. This protocol is based on imposing that for distances smaller than the correct scale the density of the data is constant. In the presented framework, to estimate the density it is necessary to know the ID, therefore, this condition is imposed self-consistently. We illustrate the usefulness and robustness of this procedure to noise by benchmarks on artificial and real-world datasets. |
| title | Scale-adaptive and robust intrinsic dimension estimation via optimal neighbourhood identification |
| topic | Machine Learning Statistics Theory Computation Methodology |
| url | https://arxiv.org/abs/2405.15132 |