Image de couverture de livre

Prompt Cache: Modular Attention Reuse for Low-Latency Inference

Fuente: arXiv

Enregistré dans:

Détails bibliographiques
Auteurs principaux:	Gim, In, Chen, Guojun, Lee, Seung-seob, Sarda, Nikhil, Khandelwal, Anurag, Zhong, Lin
Format:	Preprint
Publié:	2023
Sujets:	Computation and Language Artificial Intelligence
Accès en ligne:	Acceder al recurso
Tags:	Ajouter un tag Pas de tags, Soyez le premier à ajouter un tag!

Documents similaires

Asynchronous LLM Function Calling
par: Gim, In, et autres
Publié: (2024)

Cacheback: Speculative Decoding With Nothing But Cache
par: Ma, Zhiyao, et autres
Publié: (2025)

Pie: A Programmable Serving System for Emerging LLM Applications
par: Gim, In, et autres
Publié: (2025)

LLMCache: Layer-Wise Caching Strategies for Accelerated Reuse in Transformer Inference
par: Bansal, Harsh Vardhan
Publié: (2025)

TableCache: Primary Foreign Key Guided KV Cache Precomputation for Low Latency Text-to-SQL
par: Su, Jinbo, et autres
Publié: (2026)

FlowKV: A Disaggregated Inference Framework with Low-Latency KV Cache Transfer and Load-Aware Scheduling
par: Li, Weiqing, et autres
Publié: (2025)

CacheFocus: Dynamic Cache Re-Positioning for Efficient Retrieval-Augmented Generation
par: Lee, Kun-Hui, et autres
Publié: (2025)

From Prefix Cache to Fusion RAG Cache: Accelerating LLM Inference in Retrieval-Augmented Generation
par: Wang, Jiahao, et autres
Publié: (2026)

KVShare: An LLM Service System with Efficient and Effective Multi-Tenant KV Cache Reuse
par: Yang, Huan, et autres
Publié: (2025)

Eigen Attention: Attention in Low-Rank Space for KV Cache Compression
par: Saxena, Utkarsh, et autres
Publié: (2024)

XC-Cache: Cross-Attending to Cached Context for Efficient LLM Inference
par: Monteiro, João, et autres
Publié: (2024)

Prompt Compression in the Wild: Measuring Latency, Rate Adherence, and Quality for Faster LLM Inference
par: Kummer, Cornelius, et autres
Publié: (2026)

Generative Caching for Structurally Similar Prompts and Responses
par: Chakraborty, Sarthak, et autres
Publié: (2025)

AhaKV: Adaptive Holistic Attention-Driven KV Cache Eviction for Efficient Inference of Large Language Models
par: Gu, Yifeng, et autres
Publié: (2025)

Ethical Reasoning and Moral Value Alignment of LLMs Depend on the Language we Prompt them in
par: Agarwal, Utkarsh, et autres
Publié: (2024)

G-KV: Decoding-Time KV Cache Eviction with Global Attention
par: Liao, Mengqi, et autres
Publié: (2025)

Expected Attention: KV Cache Compression by Estimating Attention from Future Queries Distribution
par: Devoto, Alessio, et autres
Publié: (2025)

ASKD-Whisper: Adaptive Self-knowledge Distillation for Efficient and Low-Latency Automatic Speech Recognition
par: Lee, Junseok, et autres
Publié: (2026)

Serve Programs, Not Prompts
par: Gim, In, et autres
Publié: (2025)

LLMs Know What to Drop: Self-Attention Guided KV Cache Eviction for Efficient Long-Context Inference
par: Wang, Guangtao, et autres
Publié: (2025)

Sandwich Reasoning: An Answer-Reasoning-Answer Approach for Low-Latency Query Correction
par: Zhang, Chen, et autres
Publié: (2026)

Confidential Prompting: Privacy-preserving LLM Inference on Cloud
par: Li, Caihua, et autres
Publié: (2024)

HeteroCache: A Dynamic Retrieval Approach to Heterogeneous KV Cache Compression for Long-Context LLM Inference
par: Shi, Zhiyuan, et autres
Publié: (2026)

PULSE: Accelerating Distributed Pointer-Traversals on Disaggregated Memory (Extended Version)
par: Tang, Yupeng, et autres
Publié: (2023)

End-to-End Evaluation for Low-Latency Simultaneous Speech Translation
par: Huber, Christian, et autres
Publié: (2023)

SWAN: Sparse Winnowed Attention for Reduced Inference Memory via Decompression-Free KV-Cache Compression
par: S, Santhosh G, et autres
Publié: (2025)

zFLoRA: Zero-Latency Fused Low-Rank Adapters
par: Gowda, Dhananjaya, et autres
Publié: (2025)

Reformulating KV Cache Eviction Problem for Long-Context LLM Inference
par: Mai, Tho, et autres
Publié: (2026)

How and Where to Translate? The Impact of Translation Strategies in Cross-lingual LLM Prompting
par: Gupta, Aman, et autres
Publié: (2025)

Counterfactual-Consistency Prompting for Relative Temporal Understanding in Large Language Models
par: Kim, Jongho, et autres
Publié: (2025)

PIS: Linking Importance Sampling and Attention Mechanisms for Efficient Prompt Compression
par: Chen, Lizhe, et autres
Publié: (2025)

Spectral Attention Steering for Prompt Highlighting
par: Li, Weixian Waylon, et autres
Publié: (2026)

Semi-Clairvoyant Scheduling of Speculative Decoding Requests to Minimize LLM Inference Latency
par: Li, Ruixiao, et autres
Publié: (2025)

How Do Language Models Compose Functions?
par: Khandelwal, Apoorv, et autres
Publié: (2025)

CORM: Cache Optimization with Recent Message for Large Language Model Inference
par: Dai, Jincheng, et autres
Publié: (2024)

FreeKV: Boosting KV Cache Retrieval for Efficient LLM Inference
par: Liu, Guangda, et autres
Publié: (2025)

Enhancing Few-Shot Transfer Learning with Optimized Multi-Task Prompt Tuning through Modular Prompt Composition
par: Pouramini, Ahmad, et autres
Publié: (2024)

Round Attention: A Novel Round-Level Attention Mechanism to Accelerate LLM Inference
par: Tang, Yaohua, et autres
Publié: (2025)

Cross-Attention Speculative Decoding
par: Zhong, Wei, et autres
Publié: (2025)

HySparse: A Hybrid Sparse Attention Architecture with Oracle Token Selection and KV Cache Sharing
par: Gao, Yizhao, et autres
Publié: (2026)