Approximate Nearest Neighbor Search of Large Scale Vectors on Distributed Storage

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Yu, Kun, Jin, Jiabao, Zhong, Xiaoyao, Cheng, Peng, Chen, Lei, Shen, Zhitao, Song, Jingkuan, Shen, Hengtao, Lin, Xuemin
Format:	Preprint
Published:	2025
Subjects:	Databases
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

_version_	1866909857630650368
author	Yu, Kun Jin, Jiabao Zhong, Xiaoyao Cheng, Peng Chen, Lei Shen, Zhitao Song, Jingkuan Shen, Hengtao Lin, Xuemin
author_facet	Yu, Kun Jin, Jiabao Zhong, Xiaoyao Cheng, Peng Chen, Lei Shen, Zhitao Song, Jingkuan Shen, Hengtao Lin, Xuemin
contents	Approximate Nearest Neighbor Search (ANNS) in high-dimensional space is an essential operator in many online services, such as information retrieval and recommendation. Indices constructed by the state-of-the-art ANNS algorithms must be stored in single machine's memory or disk for high recall rate and throughput, suffering from substantial storage cost, constraint of limited scale and single point of failure. While distributed storage can provide a cost-effective and robust solution, there is no efficient and effective algorithms for indexing vectors in distributed storage scenarios. In this paper, we present a new graph-cluster hybrid indexing and search system which supports Distributed Storage Approximate Nearest Neighbor Search, called DSANN. DSANN can efficiently index, store, search billion-scale vector database in distributed storage and guarantee the high availability of index service. DSANN employs the concurrent index construction method to significantly reduces the complexity of index building. Then, DSANN applies Point Aggregation Graph to leverage the structural information of graph to aggregate similar vectors, optimizing storage efficiency and improving query throughput via asynchronous I/O in distributed storage. Through extensive experiments, we demonstrate DSANN can efficiently and effectively index, store and search large-scale vector datasets in distributed storage scenarios.
format	Preprint
id	arxiv_https___arxiv_org_abs_2510_17326
institution	arXiv
publishDate	2025
record_format	arxiv
spellingShingle	Approximate Nearest Neighbor Search of Large Scale Vectors on Distributed Storage Yu, Kun Jin, Jiabao Zhong, Xiaoyao Cheng, Peng Chen, Lei Shen, Zhitao Song, Jingkuan Shen, Hengtao Lin, Xuemin Databases Approximate Nearest Neighbor Search (ANNS) in high-dimensional space is an essential operator in many online services, such as information retrieval and recommendation. Indices constructed by the state-of-the-art ANNS algorithms must be stored in single machine's memory or disk for high recall rate and throughput, suffering from substantial storage cost, constraint of limited scale and single point of failure. While distributed storage can provide a cost-effective and robust solution, there is no efficient and effective algorithms for indexing vectors in distributed storage scenarios. In this paper, we present a new graph-cluster hybrid indexing and search system which supports Distributed Storage Approximate Nearest Neighbor Search, called DSANN. DSANN can efficiently index, store, search billion-scale vector database in distributed storage and guarantee the high availability of index service. DSANN employs the concurrent index construction method to significantly reduces the complexity of index building. Then, DSANN applies Point Aggregation Graph to leverage the structural information of graph to aggregate similar vectors, optimizing storage efficiency and improving query throughput via asynchronous I/O in distributed storage. Through extensive experiments, we demonstrate DSANN can efficiently and effectively index, store and search large-scale vector datasets in distributed storage scenarios.
title	Approximate Nearest Neighbor Search of Large Scale Vectors on Distributed Storage
topic	Databases
url	https://arxiv.org/abs/2510.17326