RelayGR: Scaling Long-Sequence Generative Recommendation via Cross-Stage Relay-Race Inference
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2026
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866917183467028480 |
|---|---|
| author | Wang, Jiarui Chai, Huichao Zhang, Yuanhang Zhou, Zongjin Guo, Wei Yang, Xingkun Tang, Qiang Pan, Bo Zhu, Jiawei Cheng, Ke Yan, Yuting Wang, Shulan Zhu, Yingjie Yuan, Zhengfan Huang, Jiaqi Zhang, Yuhan Sun, Xiaosong Zhang, Zhinan Zhu, Hong Zhang, Yongsheng Dong, Tiantian Xiao, Zhong Liu, Deliang Lu, Chengzhou Sun, Yuan Chen, Zhiyuan Han, Xinming Liu, Zaizhu Wang, Yaoyuan Zhang, Ziyang Liu, Yong Xu, Jinxin Sun, Yajing Yu, Zhoujun Zhou, Wenting Zhang, Qidong Zhang, Zhengyong Gu, Zhonghai Jin, Yibo Feng, Yongxiang Zuo, Pengfei |
| author_facet | Wang, Jiarui Chai, Huichao Zhang, Yuanhang Zhou, Zongjin Guo, Wei Yang, Xingkun Tang, Qiang Pan, Bo Zhu, Jiawei Cheng, Ke Yan, Yuting Wang, Shulan Zhu, Yingjie Yuan, Zhengfan Huang, Jiaqi Zhang, Yuhan Sun, Xiaosong Zhang, Zhinan Zhu, Hong Zhang, Yongsheng Dong, Tiantian Xiao, Zhong Liu, Deliang Lu, Chengzhou Sun, Yuan Chen, Zhiyuan Han, Xinming Liu, Zaizhu Wang, Yaoyuan Zhang, Ziyang Liu, Yong Xu, Jinxin Sun, Yajing Yu, Zhoujun Zhou, Wenting Zhang, Qidong Zhang, Zhengyong Gu, Zhonghai Jin, Yibo Feng, Yongxiang Zuo, Pengfei |
| contents | Real-time recommender systems execute multi-stage cascades (retrieval, pre-processing, fine-grained ranking) under strict tail-latency SLOs, leaving only tens of milliseconds for ranking. Generative recommendation (GR) models can improve quality by consuming long user-behavior sequences, but in production their online sequence length is tightly capped by the ranking-stage P99 budget. We observe that the majority of GR tokens encode user behaviors that are independent of the item candidates, suggesting an opportunity to pre-infer a user-behavior prefix once and reuse it during ranking rather than recomputing it on the critical path. Realizing this idea at industrial scale is non-trivial: the prefix cache must survive across multiple pipeline stages before the final ranking instance is determined, the user population implies cache footprints far beyond a single device, and indiscriminate pre-inference would overload shared resources under high QPS. We present RelayGR, a production system that enables in-HBM relay-race inference for GR. RelayGR selectively pre-infers long-term user prefixes, keeps their KV caches resident in HBM over the request lifecycle, and ensures the subsequent ranking can consume them without remote fetches. RelayGR combines three techniques: 1) a sequence-aware trigger that admits only at-risk requests under a bounded cache footprint and pre-inference load, 2) an affinity-aware router that co-locates cache production and consumption by routing both the auxiliary pre-infer signal and the ranking request to the same instance, and 3) a memory-aware expander that uses server-local DRAM to capture short-term cross-request reuse while avoiding redundant reloads. We implement RelayGR on Huawei Ascend NPUs and evaluate it with real queries. Under a fixed P99 SLO, RelayGR supports up to 1.5$\times$ longer sequences and improves SLO-compliant throughput by up to 3.6$\times$. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_01712 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | RelayGR: Scaling Long-Sequence Generative Recommendation via Cross-Stage Relay-Race Inference Wang, Jiarui Chai, Huichao Zhang, Yuanhang Zhou, Zongjin Guo, Wei Yang, Xingkun Tang, Qiang Pan, Bo Zhu, Jiawei Cheng, Ke Yan, Yuting Wang, Shulan Zhu, Yingjie Yuan, Zhengfan Huang, Jiaqi Zhang, Yuhan Sun, Xiaosong Zhang, Zhinan Zhu, Hong Zhang, Yongsheng Dong, Tiantian Xiao, Zhong Liu, Deliang Lu, Chengzhou Sun, Yuan Chen, Zhiyuan Han, Xinming Liu, Zaizhu Wang, Yaoyuan Zhang, Ziyang Liu, Yong Xu, Jinxin Sun, Yajing Yu, Zhoujun Zhou, Wenting Zhang, Qidong Zhang, Zhengyong Gu, Zhonghai Jin, Yibo Feng, Yongxiang Zuo, Pengfei Distributed, Parallel, and Cluster Computing Artificial Intelligence Machine Learning Real-time recommender systems execute multi-stage cascades (retrieval, pre-processing, fine-grained ranking) under strict tail-latency SLOs, leaving only tens of milliseconds for ranking. Generative recommendation (GR) models can improve quality by consuming long user-behavior sequences, but in production their online sequence length is tightly capped by the ranking-stage P99 budget. We observe that the majority of GR tokens encode user behaviors that are independent of the item candidates, suggesting an opportunity to pre-infer a user-behavior prefix once and reuse it during ranking rather than recomputing it on the critical path. Realizing this idea at industrial scale is non-trivial: the prefix cache must survive across multiple pipeline stages before the final ranking instance is determined, the user population implies cache footprints far beyond a single device, and indiscriminate pre-inference would overload shared resources under high QPS. We present RelayGR, a production system that enables in-HBM relay-race inference for GR. RelayGR selectively pre-infers long-term user prefixes, keeps their KV caches resident in HBM over the request lifecycle, and ensures the subsequent ranking can consume them without remote fetches. RelayGR combines three techniques: 1) a sequence-aware trigger that admits only at-risk requests under a bounded cache footprint and pre-inference load, 2) an affinity-aware router that co-locates cache production and consumption by routing both the auxiliary pre-infer signal and the ranking request to the same instance, and 3) a memory-aware expander that uses server-local DRAM to capture short-term cross-request reuse while avoiding redundant reloads. We implement RelayGR on Huawei Ascend NPUs and evaluate it with real queries. Under a fixed P99 SLO, RelayGR supports up to 1.5$\times$ longer sequences and improves SLO-compliant throughput by up to 3.6$\times$. |
| title | RelayGR: Scaling Long-Sequence Generative Recommendation via Cross-Stage Relay-Race Inference |
| topic | Distributed, Parallel, and Cluster Computing Artificial Intelligence Machine Learning |
| url | https://arxiv.org/abs/2601.01712 |