SilverTorch: A Unified Model-based System to Democratize Large-Scale Recommendation on GPUs
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866910199788339200 |
|---|---|
| author | Xue, Bi Wu, Hong Chen, Lei Yang, Chao Ma, Yiming Ding, Fei Wang, Zhen Wang, Liang Mao, Xiaoheng Huang, Ke Li, Xialu Xia, Peng Jian, Rui Zhao, Yanli Huang, Yanzun Deng, Yijie Tran, Harry Chang, Ryan Yu, Min Dong, Eric Wang, Jiazhou Zhang, Qianqian Zhai, Keke Yin, Hongzhang Garbacki, Pawel Zhai, Jiaqi Fang, Zheng Pan, Yiyi Ni, Min Greer, Kevin Zhang, Rui Liu, Yang |
| author_facet | Xue, Bi Wu, Hong Chen, Lei Yang, Chao Ma, Yiming Ding, Fei Wang, Zhen Wang, Liang Mao, Xiaoheng Huang, Ke Li, Xialu Xia, Peng Jian, Rui Zhao, Yanli Huang, Yanzun Deng, Yijie Tran, Harry Chang, Ryan Yu, Min Dong, Eric Wang, Jiazhou Zhang, Qianqian Zhai, Keke Yin, Hongzhang Garbacki, Pawel Zhai, Jiaqi Fang, Zheng Pan, Yiyi Ni, Min Greer, Kevin Zhang, Rui Liu, Yang |
| contents | Serving deep learning based recommendation models (DLRM) at scale is challenging. Existing approaches rely on dedicated ANN indexing and filtering services on CPUs, suffering from non-negligible costs and missing co-design opportunities. Such inefficiency makes them difficult to support complex model architectures, such as learned similarities and multi-task retrieval.
In this paper, we present SilverTorch, a model-based serving system that brings all components into one unified model. It unifies model serving by replacing standalone indexing and filtering services with model layers. We propose a model-based GPU Bloom index for feature filtering and a fused Int8 ANN kernel for nearest neighbor search. Through co-design of the ANN search and feature filtering, we reduce GPU memory usage and eliminate computation. Benefiting from this design, we scale up retrieval by introducing an OverArch scoring layer and a multi-task retrieval with a Value Model to aggregate scores. These advancements improve the retrieval accuracy and enable future studies for serving more complex models.
Our evaluation on industry-scale datasets show that SilverTorch achieves up to 23.7\times higher throughput compared to the state-of-the-art approaches. We also demonstrate that SilverTorch solution is 13.35\times more cost-efficient than CPU-based solution while improving accuracy via serving more complex models. SilverTorch is deployed at scale, serving hundreds of models online and supporting recommendation for diverse applications. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_14881 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | SilverTorch: A Unified Model-based System to Democratize Large-Scale Recommendation on GPUs Xue, Bi Wu, Hong Chen, Lei Yang, Chao Ma, Yiming Ding, Fei Wang, Zhen Wang, Liang Mao, Xiaoheng Huang, Ke Li, Xialu Xia, Peng Jian, Rui Zhao, Yanli Huang, Yanzun Deng, Yijie Tran, Harry Chang, Ryan Yu, Min Dong, Eric Wang, Jiazhou Zhang, Qianqian Zhai, Keke Yin, Hongzhang Garbacki, Pawel Zhai, Jiaqi Fang, Zheng Pan, Yiyi Ni, Min Greer, Kevin Zhang, Rui Liu, Yang Information Retrieval Serving deep learning based recommendation models (DLRM) at scale is challenging. Existing approaches rely on dedicated ANN indexing and filtering services on CPUs, suffering from non-negligible costs and missing co-design opportunities. Such inefficiency makes them difficult to support complex model architectures, such as learned similarities and multi-task retrieval. In this paper, we present SilverTorch, a model-based serving system that brings all components into one unified model. It unifies model serving by replacing standalone indexing and filtering services with model layers. We propose a model-based GPU Bloom index for feature filtering and a fused Int8 ANN kernel for nearest neighbor search. Through co-design of the ANN search and feature filtering, we reduce GPU memory usage and eliminate computation. Benefiting from this design, we scale up retrieval by introducing an OverArch scoring layer and a multi-task retrieval with a Value Model to aggregate scores. These advancements improve the retrieval accuracy and enable future studies for serving more complex models. Our evaluation on industry-scale datasets show that SilverTorch achieves up to 23.7\times higher throughput compared to the state-of-the-art approaches. We also demonstrate that SilverTorch solution is 13.35\times more cost-efficient than CPU-based solution while improving accuracy via serving more complex models. SilverTorch is deployed at scale, serving hundreds of models online and supporting recommendation for diverse applications. |
| title | SilverTorch: A Unified Model-based System to Democratize Large-Scale Recommendation on GPUs |
| topic | Information Retrieval |
| url | https://arxiv.org/abs/2511.14881 |