Cover Image

Continuous Semantic Caching for Low-Cost LLM Serving

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Atalar, Baran, Liu, Xutong, Zuo, Jinhang, Wang, Siwei, Chen, Wei, Joe-Wong, Carlee
Format:	Preprint
Published:	2026
Subjects:	Machine Learning Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Semantic Caching for Low-Cost LLM Serving: From Offline Learning to Online Adaptation
by: Liu, Xutong, et al.
Published: (2025)

Neural Bandit Based Optimal LLM Selection for a Pipeline of Subtasks
by: Atalar, Baran, et al.
Published: (2025)

Neural Combinatorial Clustered Bandits for Recommendation Systems
by: Atalar, Baran, et al.
Published: (2024)

Batch-Size Independent Regret Bounds for Combinatorial Semi-Bandits with Probabilistically Triggered Arms or Independent Arms
by: Liu, Xutong, et al.
Published: (2022)

Offline Learning for Combinatorial Multi-armed Bandits
by: Liu, Xutong, et al.
Published: (2025)

Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttention
by: Gao, Bin, et al.
Published: (2024)

DR. WELL: Dynamic Reasoning and Learning with Symbolic World Model for Embodied LLM-Based Multi-Agent Collaboration
by: Nourzad, Narjes, et al.
Published: (2025)

Intelligent Communication Planning for Constrained Environmental IoT Sensing with Reinforcement Learning
by: Hu, Yi, et al.
Published: (2023)

Efficient Contextual LLM Cascades through Budget-Constrained Policy Learning
by: Zhang, Xuechen, et al.
Published: (2024)

Tin-Tin: Towards Tiny Learning on Tiny Devices with Integer-based Neural Network Training
by: Hu, Yi, et al.
Published: (2025)

QMOS: Enhancing LLMs for Telecommunication with Question Masked loss and Option Shuffling
by: Guda, Blessed, et al.
Published: (2024)

CoRAST: Towards Foundation Model-Powered Correlated Data Analysis in Resource-Constrained CPS and IoT
by: Hu, Yi, et al.
Published: (2024)

vCache: Verified Semantic Prompt Caching
by: Schroeder, Luis Gaspar, et al.
Published: (2025)

Internal Planning in Language Models: Characterizing Horizon and Branch Awareness
by: Ustaomeroglu, Muhammed, et al.
Published: (2025)

DroidSpeak: KV Cache Sharing for Cross-LLM Communication and Multi-LLM Serving
by: Liu, Yuhan, et al.
Published: (2024)

dLLM-Cache: Accelerating Diffusion Large Language Models with Adaptive Caching
by: Liu, Zhiyuan, et al.
Published: (2025)

Emergent and Subliminal Misalignment Through the Lens of Data-Mediated Transfer
by: Askin, Baris, et al.
Published: (2026)

Contextual Combinatorial Bandits with Probabilistically Triggered Arms
by: Liu, Xutong, et al.
Published: (2023)

AttnCache: Accelerating Self-Attention Inference for LLM Prefill via Attention Cache
by: Song, Dinghong, et al.
Published: (2025)

Cost-Ordered Feasibility for Multi-Armed Bandits with Cost Subsidy
by: Juneja, Ishank, et al.
Published: (2026)

EPIC: Efficient Position-Independent Caching for Serving Large Language Models
by: Hu, Junhao, et al.
Published: (2024)

Offline Clustering of Preference Learning with Active-data Augmentation
by: Liu, Jingyuan, et al.
Published: (2025)

From Exact Hits to Close Enough: Semantic Caching for LLM Embeddings
by: Biton, Dvir David, et al.
Published: (2026)

Memory-Based Advantage Shaping for LLM-Guided Reinforcement Learning
by: Nourzad, Narjes, et al.
Published: (2026)

Pairwise Elimination with Instance-Dependent Guarantees for Bandits with Cost Subsidy
by: Juneja, Ishank, et al.
Published: (2025)

MIRA: Memory-Integrated Reinforcement Learning Agent with Limited LLM Guidance
by: Nourzad, Narjes, et al.
Published: (2026)

Coral: Cost-Efficient Multi-LLM Serving over Heterogeneous Cloud GPUs
by: Mei, Yixuan, et al.
Published: (2026)

When Are Experts Misrouted? Counterfactual Routing Analysis in Mixture-of-Experts Language Models
by: Yoon, Youngsik, et al.
Published: (2026)

SentenceKV: Efficient LLM Inference via Sentence-Level Semantic KV Caching
by: Zhu, Yuxuan, et al.
Published: (2025)

ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference
by: Sun, Hanshi, et al.
Published: (2024)

Advancing Semantic Caching for LLMs with Domain-Specific Embeddings and Synthetic Data
by: Gill, Waris, et al.
Published: (2025)

XKV: Personalized KV Cache Memory Reduction for Long-Context LLM Inference
by: Li, Weizhuo, et al.
Published: (2024)

Stochastic Bandits Robust to Adversarial Attacks
by: Wang, Xuchuang, et al.
Published: (2024)

LoLA: Low-Rank Linear Attention With Sparse Caching
by: McDermott, Luke, et al.
Published: (2025)

Cost-Efficient LLM Serving in the Cloud: VM Selection with KV Cache Offloading
by: Kim, Kihyun, et al.
Published: (2025)

AttentionPredictor: Temporal Patterns Matter for KV Cache Compression
by: Yang, Qingyue, et al.
Published: (2025)

LKV: End-to-End Learning of Head-wise Budgets and Token Selection for LLM KV Cache Eviction
by: Zhou, Enshuai, et al.
Published: (2026)

Online Multi-LLM Selection via Contextual Bandits under Unstructured Context Evolution
by: Poon, Manhin, et al.
Published: (2025)

TweakLLM: A Routing Architecture for Dynamic Tailoring of Cached Responses
by: Cheema, Muhammad Taha, et al.
Published: (2025)

AdaServe: Accelerating Multi-SLO LLM Serving with SLO-Customized Speculative Decoding
by: Li, Zikun, et al.
Published: (2025)