Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2410.12794 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916546415165440 |
|---|---|
| author | Huang, Yibo Yang, Zhenning Xing, Jiarong Dai, Yi Qiu, Yiming Wu, Dingming Lai, Fan Chen, Ang |
| author_facet | Huang, Yibo Yang, Zhenning Xing, Jiarong Dai, Yi Qiu, Yiming Wu, Dingming Lai, Fan Chen, Ang |
| contents | Efficiently serving embedding-based recommendation (EMR) models remains a significant challenge due to their increasingly large memory requirements. Today's practice splits the model across many monolithic servers, where a mix of GPUs, CPUs, and DRAM is provisioned in fixed proportions. This approach leads to suboptimal resource utilization and increased costs. Disaggregating embedding operations from neural network inference is a promising solution but raises novel networking challenges. In this paper, we discuss the design of FlexEMR for optimized EMR disaggregation. FlexEMR proposes two sets of techniques to tackle the networking challenges: Leveraging the temporal and spatial locality of embedding lookups to reduce data movement over the network, and designing an optimized multi-threaded RDMA engine for concurrent lookup subrequests. We outline the design space for each technique and present initial results from our early prototype. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_12794 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Disaggregating Embedding Recommendation Systems with FlexEMR Huang, Yibo Yang, Zhenning Xing, Jiarong Dai, Yi Qiu, Yiming Wu, Dingming Lai, Fan Chen, Ang Information Retrieval Artificial Intelligence Efficiently serving embedding-based recommendation (EMR) models remains a significant challenge due to their increasingly large memory requirements. Today's practice splits the model across many monolithic servers, where a mix of GPUs, CPUs, and DRAM is provisioned in fixed proportions. This approach leads to suboptimal resource utilization and increased costs. Disaggregating embedding operations from neural network inference is a promising solution but raises novel networking challenges. In this paper, we discuss the design of FlexEMR for optimized EMR disaggregation. FlexEMR proposes two sets of techniques to tackle the networking challenges: Leveraging the temporal and spatial locality of embedding lookups to reduce data movement over the network, and designing an optimized multi-threaded RDMA engine for concurrent lookup subrequests. We outline the design space for each technique and present initial results from our early prototype. |
| title | Disaggregating Embedding Recommendation Systems with FlexEMR |
| topic | Information Retrieval Artificial Intelligence |
| url | https://arxiv.org/abs/2410.12794 |