PerCache: Predictive Hierarchical Cache for RAG Applications on Mobile Devices
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866917207230906368 |
|---|---|
| author | Liu, Kaiwei Zeng, Liekang Xu, Lilin Yang, Bufang Yan, Zhenyu |
| author_facet | Liu, Kaiwei Zeng, Liekang Xu, Lilin Yang, Bufang Yan, Zhenyu |
| contents | Retrieval-augmented generation (RAG) has been extensively used as a de facto paradigm in various large language model (LLM)-driven applications on mobile devices, such as mobile assistants leveraging personal emails or meeting records. However, due to the lengthy prompts and the resource constraints, mobile RAG systems exhibit significantly high response latency. On this issue, one promising approach is to reuse intermediate computational results across different queries to eliminate redundant computation. But most existing approaches, such as KV cache reuse and semantic cache reuse, are designed for cloud settings and perform poorly, overlooking the distinctive characteristics of mobile RAG.
We propose PerCache, a novel hierarchical cache solution designed for reducing end-to-end latency of personalized RAG applications on mobile platforms. PerCache adopts a hierarchical architecture that progressively matches similar queries and QKV cache to maximize the reuse of intermediate results at different computing stages. To improve cache hit rate, PerCache applies a predictive method to populate cache with queries that are likely to be raised in the future. In addition, PerCache can adapt its configurations to dynamic system loads, aiming at maximizing the caching utility with minimal resource consumption. We implement PerCache on top of an existing mobile LLM inference engine with commodity mobile phones. Extensive evaluations show that PerCache can surpass the best-performing baseline by 34.4% latency reduction across various applications and maintain optimal latency performance under dynamic resource changes. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_11553 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | PerCache: Predictive Hierarchical Cache for RAG Applications on Mobile Devices Liu, Kaiwei Zeng, Liekang Xu, Lilin Yang, Bufang Yan, Zhenyu Distributed, Parallel, and Cluster Computing Retrieval-augmented generation (RAG) has been extensively used as a de facto paradigm in various large language model (LLM)-driven applications on mobile devices, such as mobile assistants leveraging personal emails or meeting records. However, due to the lengthy prompts and the resource constraints, mobile RAG systems exhibit significantly high response latency. On this issue, one promising approach is to reuse intermediate computational results across different queries to eliminate redundant computation. But most existing approaches, such as KV cache reuse and semantic cache reuse, are designed for cloud settings and perform poorly, overlooking the distinctive characteristics of mobile RAG. We propose PerCache, a novel hierarchical cache solution designed for reducing end-to-end latency of personalized RAG applications on mobile platforms. PerCache adopts a hierarchical architecture that progressively matches similar queries and QKV cache to maximize the reuse of intermediate results at different computing stages. To improve cache hit rate, PerCache applies a predictive method to populate cache with queries that are likely to be raised in the future. In addition, PerCache can adapt its configurations to dynamic system loads, aiming at maximizing the caching utility with minimal resource consumption. We implement PerCache on top of an existing mobile LLM inference engine with commodity mobile phones. Extensive evaluations show that PerCache can surpass the best-performing baseline by 34.4% latency reduction across various applications and maintain optimal latency performance under dynamic resource changes. |
| title | PerCache: Predictive Hierarchical Cache for RAG Applications on Mobile Devices |
| topic | Distributed, Parallel, and Cluster Computing |
| url | https://arxiv.org/abs/2601.11553 |