Revisiting the Index Construction of Proximity Graph-Based Approximate Nearest Neighbor Search
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866913691952218112 |
|---|---|
| author | Yang, Shuo Xie, Jiadong Liu, Yingfan Yu, Jeffrey Xu Gao, Xiyue Wang, Qianru Peng, Yanguo Cui, Jiangtao |
| author_facet | Yang, Shuo Xie, Jiadong Liu, Yingfan Yu, Jeffrey Xu Gao, Xiyue Wang, Qianru Peng, Yanguo Cui, Jiangtao |
| contents | Proximity graphs (PG) have gained increasing popularity as the state-of-the-art solutions to $k$-approximate nearest neighbor ($k$-ANN) search on high-dimensional data, which serves as a fundamental function in various fields, e.g., retrieval-augmented generation. Although PG-based approaches have the best $k$-ANN search performance, their index construction cost is superlinear to the number of points. Such superlinear cost substantially limits their scalability in the era of big data. Hence, the goal of this paper is to accelerate the construction of PG-based methods without compromising their $k$-ANN search performance.
To achieve this goal, two mainstream categories of PG are revisited: relative neighborhood graph (RNG) and navigable small world graph (NSWG). By revisiting their construction process, we find the issues of construction efficiency. To address these issues, we propose a new construction framework with a novel pruning strategy for edge selection, which accelerates RNG construction while keeping its $k$-ANN search performance. Then, we integrate this framework into NSWG construction to enhance both the construction efficiency and $k$-ANN search performance of NSWG. Extensive experiments are conducted to validate our construction framework for both RNG and NSWG, and that it significantly reduces the PG construction cost, achieving up to 5.6x speedup, while not compromising the $k$-ANN search performance. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_01231 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Revisiting the Index Construction of Proximity Graph-Based Approximate Nearest Neighbor Search Yang, Shuo Xie, Jiadong Liu, Yingfan Yu, Jeffrey Xu Gao, Xiyue Wang, Qianru Peng, Yanguo Cui, Jiangtao Databases Proximity graphs (PG) have gained increasing popularity as the state-of-the-art solutions to $k$-approximate nearest neighbor ($k$-ANN) search on high-dimensional data, which serves as a fundamental function in various fields, e.g., retrieval-augmented generation. Although PG-based approaches have the best $k$-ANN search performance, their index construction cost is superlinear to the number of points. Such superlinear cost substantially limits their scalability in the era of big data. Hence, the goal of this paper is to accelerate the construction of PG-based methods without compromising their $k$-ANN search performance. To achieve this goal, two mainstream categories of PG are revisited: relative neighborhood graph (RNG) and navigable small world graph (NSWG). By revisiting their construction process, we find the issues of construction efficiency. To address these issues, we propose a new construction framework with a novel pruning strategy for edge selection, which accelerates RNG construction while keeping its $k$-ANN search performance. Then, we integrate this framework into NSWG construction to enhance both the construction efficiency and $k$-ANN search performance of NSWG. Extensive experiments are conducted to validate our construction framework for both RNG and NSWG, and that it significantly reduces the PG construction cost, achieving up to 5.6x speedup, while not compromising the $k$-ANN search performance. |
| title | Revisiting the Index Construction of Proximity Graph-Based Approximate Nearest Neighbor Search |
| topic | Databases |
| url | https://arxiv.org/abs/2410.01231 |