Cover Image

LLM Cache Bandit Revisited: Addressing Query Heterogeneity for Cost-Effective LLM Inference

Fuente: arXiv

Saved in:

Bibliographic Details
Main Authors:	Yang, Hantao, Xie, Hong, Lian, Defu, Chen, Enhong
Format:	Preprint
Published:	2025
Subjects:	Computation and Language
Online Access:	Acceder al recurso
Tags:	Add Tag No Tags, Be the first to tag this record!

Similar Items

Federated Contextual Cascading Bandits with Asynchronous Communication and Heterogeneous Users
by: Yang, Hantao, et al.
Published: (2024)

IE as Cache: Information Extraction Enhanced Agentic Reasoning
by: Lv, Hang, et al.
Published: (2026)

HumanLLM: Towards Personalized Understanding and Simulation of Human Nature
by: Lei, Yuxuan, et al.
Published: (2026)

Taming the Fragility of KV Cache Eviction in LLM Inference
by: Feng, Yuan, et al.
Published: (2025)

Understanding the planning of LLM agents: A survey
by: Huang, Xu, et al.
Published: (2024)

Scaling Federated Linear Contextual Bandits via Sketching
by: Yang, Hantao, et al.
Published: (2026)

HeteroCache: A Dynamic Retrieval Approach to Heterogeneous KV Cache Compression for Long-Context LLM Inference
by: Shi, Zhiyuan, et al.
Published: (2026)

Entropy Law: The Story Behind Data Compression and LLM Performance
by: Yin, Mingjia, et al.
Published: (2024)

Continuous Semantic Caching for Low-Cost LLM Serving
by: Atalar, Baran, et al.
Published: (2026)

From Prefix Cache to Fusion RAG Cache: Accelerating LLM Inference in Retrieval-Augmented Generation
by: Wang, Jiahao, et al.
Published: (2026)

Learning to Substitute Components for Compositional Generalization
by: Li, Zhaoyi, et al.
Published: (2025)

XC-Cache: Cross-Attending to Cached Context for Efficient LLM Inference
by: Monteiro, João, et al.
Published: (2024)

DASH-KV: Accelerating Long-Context LLM Inference via Asymmetric KV Cache Hashing
by: Guo, Jinyu, et al.
Published: (2026)

Efficient Long-Context LLM Inference via KV Cache Clustering
by: Hu, Jie, et al.
Published: (2025)

AttnCache: Accelerating Self-Attention Inference for LLM Prefill via Attention Cache
by: Song, Dinghong, et al.
Published: (2025)

PyramidInfer: Pyramid KV Cache Compression for High-throughput LLM Inference
by: Yang, Dongjie, et al.
Published: (2024)

EquivPruner: Boosting Efficiency and Quality in LLM-Based Search via Action Pruning
by: Liu, Jiawei, et al.
Published: (2025)

Mitigating KV Cache Competition to Enhance User Experience in LLM Inference
by: Shen, Haiying, et al.
Published: (2025)

Compressing KV Cache for Long-Context LLM Inference with Inter-Layer Attention Similarity
by: Ma, Da, et al.
Published: (2024)

Mask Tokens as Prophet: Fine-Grained Cache Eviction for Efficient dLLM Inference
by: Huang, Jianuo, et al.
Published: (2025)

Understanding Privacy Risks of Embeddings Induced by Large Language Models
by: Zhu, Zhihao, et al.
Published: (2024)

InstInfer: In-Storage Attention Offloading for Cost-Effective Long-Context LLM Inference
by: Pan, Xiurui, et al.
Published: (2024)

KVDrive: A Holistic Multi-Tier KV Cache Management System for Long-Context LLM Inference
by: Lin, Jian, et al.
Published: (2026)

Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inference
by: Feng, Yuan, et al.
Published: (2024)

EvolKV: Evolutionary KV Cache Compression for LLM Inference
by: Yu, Bohan, et al.
Published: (2025)

Retrieve-Plan-Generation: An Iterative Planning and Answering Framework for Knowledge-Intensive LLM Generation
by: Lyu, Yuanjie, et al.
Published: (2024)

Keep the Cost Down: A Review on Methods to Optimize LLM' s KV-Cache Consumption
by: Shi, Luohe, et al.
Published: (2024)

On Cost-Effective LLM-as-a-Judge Improvement Techniques
by: Lail, Ryan, et al.
Published: (2026)

Multi-agent Multi-armed Bandits with Stochastic Sharable Arm Capacities
by: Xie, Hong, et al.
Published: (2024)

Mitigating the Language Mismatch and Repetition Issues in LLM-based Machine Translation via Model Editing
by: Wang, Weichuan, et al.
Published: (2024)

TTKV: Temporal-Tiered KV Cache for Long-Context LLM Inference
by: Dzikanyanga, Gradwell, et al.
Published: (2026)

Reformulating KV Cache Eviction Problem for Long-Context LLM Inference
by: Mai, Tho, et al.
Published: (2026)

Using LLM to select the right SQL Query from candidates
by: Li, Zhenwen, et al.
Published: (2024)

Learning from Emptiness: De-biasing Listwise Rerankers with Content-Agnostic Probability Calibration
by: Lv, Hang, et al.
Published: (2026)

WindowKV: Task-Adaptive Group-Wise KV Cache Window Selection for Efficient LLM Inference
by: Zuo, Youhui, et al.
Published: (2025)

KV Cache Transform Coding for Compact Storage in LLM Inference
by: Staniszewski, Konrad, et al.
Published: (2025)

KVShare: An LLM Service System with Efficient and Effective Multi-Tenant KV Cache Reuse
by: Yang, Huan, et al.
Published: (2025)

UniMEL: A Unified Framework for Multimodal Entity Linking with Large Language Models
by: Qi, Liu, et al.
Published: (2024)

ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inference
by: Sun, Hanshi, et al.
Published: (2024)

BUZZ: Beehive-structured Sparse KV Cache with Segmented Heavy Hitters for Efficient LLM Inference
by: Zhao, Junqi, et al.
Published: (2024)