Guardado en:
| Autores principales: | , , , , , , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2026
|
| Materias: | |
| Acceso en línea: | https://arxiv.org/abs/2604.22881 |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866908991416696832 |
|---|---|
| author | Wang, Xin Ma, Chi Chen, Shaobin Wang, Pu Zhou, Menglei Qiu, Junyi Chen, Qiaorui Sun, Jiayu Liu, Shijie Wang, Zehuan Yu, Lei Liu, Chuan Jiang, Fei Lin, Wei Wang, Hao Jiang, Jiawei Yan, Xiao |
| author_facet | Wang, Xin Ma, Chi Chen, Shaobin Wang, Pu Zhou, Menglei Qiu, Junyi Chen, Qiaorui Sun, Jiayu Liu, Shijie Wang, Zehuan Yu, Lei Liu, Chuan Jiang, Fei Lin, Wei Wang, Hao Jiang, Jiawei Yan, Xiao |
| contents | Generative recommendation (GR) offers superior modeling capabilities but suffers from prohibitive inference costs due to the repeated encoding of long user histories. While cross-request Key-Value (KV) cache reuse presents a significant optimization opportunity, the massive scale of individual user states creates a storage explosion that far exceeds physical GPU limits. We propose MTServe, a hierarchical cache management system that virtualizes GPU memory by leveraging host RAM as a scalable backup store. To bridge the I/O gap between tiers, MTServe introduces a suite of system-level optimizations, including a hybrid storage layout, an asynchronous data transfer pipeline, and a locality-driven replacement policy. On both public and production datasets, MTServe delivers up to 3.1* speedup while maintaining near-perfect hit ratios (>98.5%). |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_22881 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | MTServe: Efficient Serving for Generative Recommendation Models with Hierarchical Caches Wang, Xin Ma, Chi Chen, Shaobin Wang, Pu Zhou, Menglei Qiu, Junyi Chen, Qiaorui Sun, Jiayu Liu, Shijie Wang, Zehuan Yu, Lei Liu, Chuan Jiang, Fei Lin, Wei Wang, Hao Jiang, Jiawei Yan, Xiao Machine Learning Artificial Intelligence Generative recommendation (GR) offers superior modeling capabilities but suffers from prohibitive inference costs due to the repeated encoding of long user histories. While cross-request Key-Value (KV) cache reuse presents a significant optimization opportunity, the massive scale of individual user states creates a storage explosion that far exceeds physical GPU limits. We propose MTServe, a hierarchical cache management system that virtualizes GPU memory by leveraging host RAM as a scalable backup store. To bridge the I/O gap between tiers, MTServe introduces a suite of system-level optimizations, including a hybrid storage layout, an asynchronous data transfer pipeline, and a locality-driven replacement policy. On both public and production datasets, MTServe delivers up to 3.1* speedup while maintaining near-perfect hit ratios (>98.5%). |
| title | MTServe: Efficient Serving for Generative Recommendation Models with Hierarchical Caches |
| topic | Machine Learning Artificial Intelligence |
| url | https://arxiv.org/abs/2604.22881 |