Semantic Caching of Contextual Summaries for Efficient Question-Answering with Language Models
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866910947905372160 |
|---|---|
| author | Couturier, Camille Mastorakis, Spyros Shen, Haiying Rajmohan, Saravan Rühle, Victor |
| author_facet | Couturier, Camille Mastorakis, Spyros Shen, Haiying Rajmohan, Saravan Rühle, Victor |
| contents | Large Language Models (LLMs) are increasingly deployed across edge and cloud platforms for real-time question-answering and retrieval-augmented generation. However, processing lengthy contexts in distributed systems incurs high computational overhead, memory usage, and network bandwidth. This paper introduces a novel semantic caching approach for storing and reusing intermediate contextual summaries, enabling efficient information reuse across similar queries in LLM-based QA workflows. Our method reduces redundant computations by up to 50-60% while maintaining answer accuracy comparable to full document processing, as demonstrated on NaturalQuestions, TriviaQA, and a synthetic ArXiv dataset. This approach balances computational cost and response quality, critical for real-time AI assistants. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_11271 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Semantic Caching of Contextual Summaries for Efficient Question-Answering with Language Models Couturier, Camille Mastorakis, Spyros Shen, Haiying Rajmohan, Saravan Rühle, Victor Computation and Language Artificial Intelligence Information Retrieval Machine Learning I.2.7 Large Language Models (LLMs) are increasingly deployed across edge and cloud platforms for real-time question-answering and retrieval-augmented generation. However, processing lengthy contexts in distributed systems incurs high computational overhead, memory usage, and network bandwidth. This paper introduces a novel semantic caching approach for storing and reusing intermediate contextual summaries, enabling efficient information reuse across similar queries in LLM-based QA workflows. Our method reduces redundant computations by up to 50-60% while maintaining answer accuracy comparable to full document processing, as demonstrated on NaturalQuestions, TriviaQA, and a synthetic ArXiv dataset. This approach balances computational cost and response quality, critical for real-time AI assistants. |
| title | Semantic Caching of Contextual Summaries for Efficient Question-Answering with Language Models |
| topic | Computation and Language Artificial Intelligence Information Retrieval Machine Learning I.2.7 |
| url | https://arxiv.org/abs/2505.11271 |