vCache: Verified Semantic Prompt Caching
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866908844811091968 |
|---|---|
| author | Schroeder, Luis Gaspar Desai, Aditya Cuadron, Alejandro Chu, Kyle Liu, Shu Zhao, Mark Krusche, Stephan Kemper, Alfons Zaharia, Matei Gonzalez, Joseph E. |
| author_facet | Schroeder, Luis Gaspar Desai, Aditya Cuadron, Alejandro Chu, Kyle Liu, Shu Zhao, Mark Krusche, Stephan Kemper, Alfons Zaharia, Matei Gonzalez, Joseph E. |
| contents | Semantic caches return cached responses for semantically similar prompts to reduce LLM inference latency and cost. They embed cached prompts and store them alongside their response in a vector database. Embedding similarity metrics assign a numerical score to quantify the similarity between a request and its nearest neighbor prompt from the cache. Existing systems use the same static similarity threshold across all requests to determine whether two prompts can share similar responses. However, we observe that static thresholds do not give formal correctness guarantees, result in unexpected error rates, and lead to suboptimal cache hit rates. This paper proposes vCache, the first verified semantic cache with user-defined error rate guarantees for predictable performance. It employs an online learning algorithm to estimate an optimal threshold for each cached prompt, enabling reliable cache responses without additional training. Our experiments show that vCache consistently meets the specified error bounds while outperforming state-of-the-art static-threshold and fine-tuned embedding baselines with up to 12.5$\times$ higher cache hit and 26$\times$ lower error rates. We release the vCache implementation and four benchmarks to support future research. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_03771 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | vCache: Verified Semantic Prompt Caching Schroeder, Luis Gaspar Desai, Aditya Cuadron, Alejandro Chu, Kyle Liu, Shu Zhao, Mark Krusche, Stephan Kemper, Alfons Zaharia, Matei Gonzalez, Joseph E. Machine Learning Computation and Language Semantic caches return cached responses for semantically similar prompts to reduce LLM inference latency and cost. They embed cached prompts and store them alongside their response in a vector database. Embedding similarity metrics assign a numerical score to quantify the similarity between a request and its nearest neighbor prompt from the cache. Existing systems use the same static similarity threshold across all requests to determine whether two prompts can share similar responses. However, we observe that static thresholds do not give formal correctness guarantees, result in unexpected error rates, and lead to suboptimal cache hit rates. This paper proposes vCache, the first verified semantic cache with user-defined error rate guarantees for predictable performance. It employs an online learning algorithm to estimate an optimal threshold for each cached prompt, enabling reliable cache responses without additional training. Our experiments show that vCache consistently meets the specified error bounds while outperforming state-of-the-art static-threshold and fine-tuned embedding baselines with up to 12.5$\times$ higher cache hit and 26$\times$ lower error rates. We release the vCache implementation and four benchmarks to support future research. |
| title | vCache: Verified Semantic Prompt Caching |
| topic | Machine Learning Computation and Language |
| url | https://arxiv.org/abs/2502.03771 |