Table of Contents: :: Library Catalog

Saved in:

Bibliographic Details
Main Authors:	Wang, Xin, Ma, Chi, Chen, Shaobin, Wang, Pu, Zhou, Menglei, Qiu, Junyi, Chen, Qiaorui, Sun, Jiayu, Liu, Shijie, Wang, Zehuan, Yu, Lei, Liu, Chuan, Jiang, Fei, Lin, Wei, Wang, Hao, Jiang, Jiawei, Yan, Xiao
Format:	Preprint
Published:	2026
Subjects:	Machine Learning Artificial Intelligence
Online Access:	https://arxiv.org/abs/2604.22881
Tags:	Add Tag No Tags, Be the first to tag this record!

Table of Contents:

Generative recommendation (GR) offers superior modeling capabilities but suffers from prohibitive inference costs due to the repeated encoding of long user histories. While cross-request Key-Value (KV) cache reuse presents a significant optimization opportunity, the massive scale of individual user states creates a storage explosion that far exceeds physical GPU limits. We propose MTServe, a hierarchical cache management system that virtualizes GPU memory by leveraging host RAM as a scalable backup store. To bridge the I/O gap between tiers, MTServe introduces a suite of system-level optimizations, including a hybrid storage layout, an asynchronous data transfer pipeline, and a locality-driven replacement policy. On both public and production datasets, MTServe delivers up to 3.1* speedup while maintaining near-perfect hit ratios (>98.5%).

Similar Items