A Highly Efficient Cross-matching Scheme using Learned Index Structure
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866908320393068544 |
|---|---|
| author | Lam, Phu-Minh Fan, Dongwei Wei, Hongbo Wang, Jun Zhou, Yu Ma, Qi Zhang, Baolong Zhang, Xiazhao Wang, Yongheng |
| author_facet | Lam, Phu-Minh Fan, Dongwei Wei, Hongbo Wang, Jun Zhou, Yu Ma, Qi Zhang, Baolong Zhang, Xiazhao Wang, Yongheng |
| contents | Spatial data fusion is a bottleneck when it meets the scale of 10 billion records. Cross-matching celestial catalogs is just one example of this. To challenge this, we present a framework that enables efficient cross-matching using Learned Index Structures. Our approach involves a data transformation method to map multi-dimensional data into easily learnable distributions, coupled with a novel search algorithm that leverages the advantages of model pairs, significantly enhancing the efficiency of nearest-neighbor search. In this study, we utilized celestial catalog data derived from astronomical surveys to construct the index and evaluated the speed of the cross-matching process. Using the HEALPix segmentation scheme, we built an independent model object for each tile and developed an end-to-end pipeline to construct a framework with semantic guarantees for record retrieval in query and range search. Our results show that the proposed method improves cross-matching speed by more than four times compared to KD-trees for a radius range between 1 milli-arcseconds and 100 arcseconds. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_10931 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | A Highly Efficient Cross-matching Scheme using Learned Index Structure Lam, Phu-Minh Fan, Dongwei Wei, Hongbo Wang, Jun Zhou, Yu Ma, Qi Zhang, Baolong Zhang, Xiazhao Wang, Yongheng Instrumentation and Methods for Astrophysics Spatial data fusion is a bottleneck when it meets the scale of 10 billion records. Cross-matching celestial catalogs is just one example of this. To challenge this, we present a framework that enables efficient cross-matching using Learned Index Structures. Our approach involves a data transformation method to map multi-dimensional data into easily learnable distributions, coupled with a novel search algorithm that leverages the advantages of model pairs, significantly enhancing the efficiency of nearest-neighbor search. In this study, we utilized celestial catalog data derived from astronomical surveys to construct the index and evaluated the speed of the cross-matching process. Using the HEALPix segmentation scheme, we built an independent model object for each tile and developed an end-to-end pipeline to construct a framework with semantic guarantees for record retrieval in query and range search. Our results show that the proposed method improves cross-matching speed by more than four times compared to KD-trees for a radius range between 1 milli-arcseconds and 100 arcseconds. |
| title | A Highly Efficient Cross-matching Scheme using Learned Index Structure |
| topic | Instrumentation and Methods for Astrophysics |
| url | https://arxiv.org/abs/2504.10931 |