Saved in:
| Main Author: | |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2602.22214 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911469381091328 |
|---|---|
| author | Calin, Teodor-Ioan |
| author_facet | Calin, Teodor-Ioan |
| contents | High-dimensional similarity search underpins modern retrieval systems, yet uniform search strategies fail to exploit the heterogeneous nature of real-world query distributions. We present an adaptive prefiltering framework that leverages query frequency patterns and cluster coherence metrics to dynamically allocate computational budgets. Our approach partitions the query space into frequency tiers following Zipfian distributions and assigns differentiated search policies based on historical access patterns and local density characteristics. Experiments on ImageNet-1k using CLIP embeddings demonstrate that frequency-aware budget allocation achieves equivalent recall with 20.4% fewer distance computations compared to static nprobe selection, while maintaining sub-millisecond latency on GPU-accelerated FAISS indices. The framework introduces minimal overhead through lightweight frequency tracking and provides graceful degradation for unseen queries through coherence-based fallback policies. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2602_22214 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Adaptive Prefiltering for High-Dimensional Similarity Search: A Frequency-Aware Approach Calin, Teodor-Ioan Information Retrieval Computer Vision and Pattern Recognition H.3.3; I.5.3; H.2.4 High-dimensional similarity search underpins modern retrieval systems, yet uniform search strategies fail to exploit the heterogeneous nature of real-world query distributions. We present an adaptive prefiltering framework that leverages query frequency patterns and cluster coherence metrics to dynamically allocate computational budgets. Our approach partitions the query space into frequency tiers following Zipfian distributions and assigns differentiated search policies based on historical access patterns and local density characteristics. Experiments on ImageNet-1k using CLIP embeddings demonstrate that frequency-aware budget allocation achieves equivalent recall with 20.4% fewer distance computations compared to static nprobe selection, while maintaining sub-millisecond latency on GPU-accelerated FAISS indices. The framework introduces minimal overhead through lightweight frequency tracking and provides graceful degradation for unseen queries through coherence-based fallback policies. |
| title | Adaptive Prefiltering for High-Dimensional Similarity Search: A Frequency-Aware Approach |
| topic | Information Retrieval Computer Vision and Pattern Recognition H.3.3; I.5.3; H.2.4 |
| url | https://arxiv.org/abs/2602.22214 |