Incremental IVF Index Maintenance for Streaming Vector Search
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866916465801691136 |
|---|---|
| author | Mohoney, Jason Pacaci, Anil Chowdhury, Shihabur Rahman Minhas, Umar Farooq Pound, Jeffery Renggli, Cedric Reyhani, Nima Ilyas, Ihab F. Rekatsinas, Theodoros Venkataraman, Shivaram |
| author_facet | Mohoney, Jason Pacaci, Anil Chowdhury, Shihabur Rahman Minhas, Umar Farooq Pound, Jeffery Renggli, Cedric Reyhani, Nima Ilyas, Ihab F. Rekatsinas, Theodoros Venkataraman, Shivaram |
| contents | The prevalence of vector similarity search in modern machine learning applications and the continuously changing nature of data processed by these applications necessitate efficient and effective index maintenance techniques for vector search indexes. Designed primarily for static workloads, existing vector search indexes degrade in search quality and performance as the underlying data is updated unless costly index reconstruction is performed. To address this, we introduce Ada-IVF, an incremental indexing methodology for Inverted File (IVF) indexes. Ada-IVF consists of 1) an adaptive maintenance policy that decides which index partitions are problematic for performance and should be repartitioned and 2) a local re-clustering mechanism that determines how to repartition them. Compared with state-of-the-art dynamic IVF index maintenance strategies, Ada-IVF achieves an average of 2x and up to 5x higher update throughput across a range of benchmark workloads. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_00970 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Incremental IVF Index Maintenance for Streaming Vector Search Mohoney, Jason Pacaci, Anil Chowdhury, Shihabur Rahman Minhas, Umar Farooq Pound, Jeffery Renggli, Cedric Reyhani, Nima Ilyas, Ihab F. Rekatsinas, Theodoros Venkataraman, Shivaram Databases Artificial Intelligence Machine Learning The prevalence of vector similarity search in modern machine learning applications and the continuously changing nature of data processed by these applications necessitate efficient and effective index maintenance techniques for vector search indexes. Designed primarily for static workloads, existing vector search indexes degrade in search quality and performance as the underlying data is updated unless costly index reconstruction is performed. To address this, we introduce Ada-IVF, an incremental indexing methodology for Inverted File (IVF) indexes. Ada-IVF consists of 1) an adaptive maintenance policy that decides which index partitions are problematic for performance and should be repartitioned and 2) a local re-clustering mechanism that determines how to repartition them. Compared with state-of-the-art dynamic IVF index maintenance strategies, Ada-IVF achieves an average of 2x and up to 5x higher update throughput across a range of benchmark workloads. |
| title | Incremental IVF Index Maintenance for Streaming Vector Search |
| topic | Databases Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2411.00970 |