TableCache: Primary Foreign Key Guided KV Cache Precomputation for Low Latency Text-to-SQL
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2026
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866914252079497216 |
|---|---|
| author | Su, Jinbo Hu, Yuxuan Li, Cuiping Chen, Hong Li, Jia Ma, Lintao Zhang, Jing |
| author_facet | Su, Jinbo Hu, Yuxuan Li, Cuiping Chen, Hong Li, Jia Ma, Lintao Zhang, Jing |
| contents | In Text-to-SQL tasks, existing LLM-based methods often include extensive database schemas in prompts, leading to long context lengths and increased prefilling latency. While user queries typically focus on recurrent table sets-offering an opportunity for KV cache sharing across queries-current inference engines, such as SGLang and vLLM, generate redundant prefix cache copies when processing user queries with varying table orders. To address this inefficiency, we propose precomputing table representations as KV caches offline and querying the required ones online. A key aspect of our approach is the computation of table caches while preserving primary foreign key relationships between tables. Additionally, we construct a Table Trie structure to facilitate efficient KV cache lookups during inference. To enhance cache performance, we introduce a cache management system with a query reranking strategy to improve cache hit rates and a computation loading pipeline for parallelizing model inference and cache loading. Experimental results show that our proposed TableCache achieves up to a 3.62x speedup in Time to First Token (TTFT) with negligible performance degradation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_08743 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | TableCache: Primary Foreign Key Guided KV Cache Precomputation for Low Latency Text-to-SQL Su, Jinbo Hu, Yuxuan Li, Cuiping Chen, Hong Li, Jia Ma, Lintao Zhang, Jing Computation and Language Artificial Intelligence In Text-to-SQL tasks, existing LLM-based methods often include extensive database schemas in prompts, leading to long context lengths and increased prefilling latency. While user queries typically focus on recurrent table sets-offering an opportunity for KV cache sharing across queries-current inference engines, such as SGLang and vLLM, generate redundant prefix cache copies when processing user queries with varying table orders. To address this inefficiency, we propose precomputing table representations as KV caches offline and querying the required ones online. A key aspect of our approach is the computation of table caches while preserving primary foreign key relationships between tables. Additionally, we construct a Table Trie structure to facilitate efficient KV cache lookups during inference. To enhance cache performance, we introduce a cache management system with a query reranking strategy to improve cache hit rates and a computation loading pipeline for parallelizing model inference and cache loading. Experimental results show that our proposed TableCache achieves up to a 3.62x speedup in Time to First Token (TTFT) with negligible performance degradation. |
| title | TableCache: Primary Foreign Key Guided KV Cache Precomputation for Low Latency Text-to-SQL |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2601.08743 |