SparKV: Overhead-Aware KV Cache Loading for Efficient On-Device LLM Inference
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2026
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866909012993245184 |
|---|---|
| author | Liu, Hongyao Zhai, Liuqun Wang, Junyi Fang, Zhengru |
| author_facet | Liu, Hongyao Zhai, Liuqun Wang, Junyi Fang, Zhengru |
| contents | Efficient inference for on-device Large Language Models (LLMs) remains challenging due to limited hardware resources and the high cost of the prefill stage, which processes the full input context to construct Key-Value (KV) caches. We present SparKV, an adaptive KV loading framework that combines cloud-based KV streaming with on-device computation. SparKV models the cost of individual KV chunks and decides whether each chunk should be streamed or computed locally, while overlapping the two execution paths to reduce latency. To handle fluctuations in wireless connectivity and edge resource availability, SparKV further refines offline-generated schedules at runtime to rebalance communication and computation costs. Experiments across diverse datasets, LLMs, and edge devices show that SparKV reduces Time-to-First-Token by 1.3$x-5.1x with negligible impact on response quality, while lowering per-request energy consumption by 1.5x to 3.3x, demonstrating its robustness and practicality for real-world on-device deployment. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_21231 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | SparKV: Overhead-Aware KV Cache Loading for Efficient On-Device LLM Inference Liu, Hongyao Zhai, Liuqun Wang, Junyi Fang, Zhengru Networking and Internet Architecture Artificial Intelligence Performance Efficient inference for on-device Large Language Models (LLMs) remains challenging due to limited hardware resources and the high cost of the prefill stage, which processes the full input context to construct Key-Value (KV) caches. We present SparKV, an adaptive KV loading framework that combines cloud-based KV streaming with on-device computation. SparKV models the cost of individual KV chunks and decides whether each chunk should be streamed or computed locally, while overlapping the two execution paths to reduce latency. To handle fluctuations in wireless connectivity and edge resource availability, SparKV further refines offline-generated schedules at runtime to rebalance communication and computation costs. Experiments across diverse datasets, LLMs, and edge devices show that SparKV reduces Time-to-First-Token by 1.3$x-5.1x with negligible impact on response quality, while lowering per-request energy consumption by 1.5x to 3.3x, demonstrating its robustness and practicality for real-world on-device deployment. |
| title | SparKV: Overhead-Aware KV Cache Loading for Efficient On-Device LLM Inference |
| topic | Networking and Internet Architecture Artificial Intelligence Performance |
| url | https://arxiv.org/abs/2604.21231 |