Exploring Distributed Vector Databases Performance on HPC Platforms: A Study with Qdrant
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866915558321029120 |
|---|---|
| author | Ockerman, Seth Gueroudji, Amal Oh, Song Young Underwood, Robert Chia, Nicholas Chard, Kyle Ross, Robert Venkataraman, Shivaram |
| author_facet | Ockerman, Seth Gueroudji, Amal Oh, Song Young Underwood, Robert Chia, Nicholas Chard, Kyle Ross, Robert Venkataraman, Shivaram |
| contents | Vector databases have rapidly grown in popularity, enabling efficient similarity search over data such as text, images, and video. They now play a central role in modern AI workflows, aiding large language models by grounding model outputs in external literature through retrieval-augmented generation. Despite their importance, little is known about the performance characteristics of vector databases in high-performance computing (HPC) systems that drive large-scale science. This work presents an empirical study of distributed vector database performance on the Polaris supercomputer in the Argonne Leadership Computing Facility. We construct a realistic biological-text workload from BV-BRC and generate embeddings from the peS2o corpus using Qwen3-Embedding-4B. We select Qdrant to evaluate insertion, index construction, and query latency with up to 32 workers. Informed by practical lessons from our experience, this work takes a first step toward characterizing vector database performance on HPC platforms to guide future research and optimization. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_12384 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Exploring Distributed Vector Databases Performance on HPC Platforms: A Study with Qdrant Ockerman, Seth Gueroudji, Amal Oh, Song Young Underwood, Robert Chia, Nicholas Chard, Kyle Ross, Robert Venkataraman, Shivaram Distributed, Parallel, and Cluster Computing Databases Vector databases have rapidly grown in popularity, enabling efficient similarity search over data such as text, images, and video. They now play a central role in modern AI workflows, aiding large language models by grounding model outputs in external literature through retrieval-augmented generation. Despite their importance, little is known about the performance characteristics of vector databases in high-performance computing (HPC) systems that drive large-scale science. This work presents an empirical study of distributed vector database performance on the Polaris supercomputer in the Argonne Leadership Computing Facility. We construct a realistic biological-text workload from BV-BRC and generate embeddings from the peS2o corpus using Qwen3-Embedding-4B. We select Qdrant to evaluate insertion, index construction, and query latency with up to 32 workers. Informed by practical lessons from our experience, this work takes a first step toward characterizing vector database performance on HPC platforms to guide future research and optimization. |
| title | Exploring Distributed Vector Databases Performance on HPC Platforms: A Study with Qdrant |
| topic | Distributed, Parallel, and Cluster Computing Databases |
| url | https://arxiv.org/abs/2509.12384 |