DLRover-RM: Resource Optimization for Deep Recommendation Models Training in the Cloud
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2023
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866929402357481472 |
|---|---|
| author | Wang, Qinlong Lan, Tingfeng Tang, Yinghao Huang, Ziling Du, Yiheng Zhang, Haitao Sha, Jian Lu, Hui Zhou, Yuanchun Zhang, Ke Tang, Mingjie |
| author_facet | Wang, Qinlong Lan, Tingfeng Tang, Yinghao Huang, Ziling Du, Yiheng Zhang, Haitao Sha, Jian Lu, Hui Zhou, Yuanchun Zhang, Ke Tang, Mingjie |
| contents | Deep learning recommendation models (DLRM) rely on large embedding tables to manage categorical sparse features. Expanding such embedding tables can significantly enhance model performance, but at the cost of increased GPU/CPU/memory usage. Meanwhile, tech companies have built extensive cloud-based services to accelerate training DLRM models at scale. In this paper, we conduct a deep investigation of the DLRM training platforms at AntGroup and reveal two critical challenges: low resource utilization due to suboptimal configurations by users and the tendency to encounter abnormalities due to an unstable cloud environment. To overcome them, we introduce DLRover-RM, an elastic training framework for DLRMs designed to increase resource utilization and handle the instability of a cloud environment. DLRover-RM develops a resource-performance model by considering the unique characteristics of DLRMs and a three-stage heuristic strategy to automatically allocate and dynamically adjust resources for DLRM training jobs for higher resource utilization. Further, DLRover-RM develops multiple mechanisms to ensure efficient and reliable execution of DLRM training jobs. Our extensive evaluation shows that DLRover-RM reduces job completion times by 31%, increases the job completion rate by 6%, enhances CPU usage by 15%, and improves memory utilization by 20%, compared to state-of-the-art resource scheduling frameworks. DLRover-RM has been widely deployed at AntGroup and processes thousands of DLRM training jobs on a daily basis. DLRover-RM is open-sourced and has been adopted by 10+ companies. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2304_01468 |
| institution | arXiv |
| publishDate | 2023 |
| record_format | arxiv |
| spellingShingle | DLRover-RM: Resource Optimization for Deep Recommendation Models Training in the Cloud Wang, Qinlong Lan, Tingfeng Tang, Yinghao Huang, Ziling Du, Yiheng Zhang, Haitao Sha, Jian Lu, Hui Zhou, Yuanchun Zhang, Ke Tang, Mingjie Distributed, Parallel, and Cluster Computing Artificial Intelligence Databases Deep learning recommendation models (DLRM) rely on large embedding tables to manage categorical sparse features. Expanding such embedding tables can significantly enhance model performance, but at the cost of increased GPU/CPU/memory usage. Meanwhile, tech companies have built extensive cloud-based services to accelerate training DLRM models at scale. In this paper, we conduct a deep investigation of the DLRM training platforms at AntGroup and reveal two critical challenges: low resource utilization due to suboptimal configurations by users and the tendency to encounter abnormalities due to an unstable cloud environment. To overcome them, we introduce DLRover-RM, an elastic training framework for DLRMs designed to increase resource utilization and handle the instability of a cloud environment. DLRover-RM develops a resource-performance model by considering the unique characteristics of DLRMs and a three-stage heuristic strategy to automatically allocate and dynamically adjust resources for DLRM training jobs for higher resource utilization. Further, DLRover-RM develops multiple mechanisms to ensure efficient and reliable execution of DLRM training jobs. Our extensive evaluation shows that DLRover-RM reduces job completion times by 31%, increases the job completion rate by 6%, enhances CPU usage by 15%, and improves memory utilization by 20%, compared to state-of-the-art resource scheduling frameworks. DLRover-RM has been widely deployed at AntGroup and processes thousands of DLRM training jobs on a daily basis. DLRover-RM is open-sourced and has been adopted by 10+ companies. |
| title | DLRover-RM: Resource Optimization for Deep Recommendation Models Training in the Cloud |
| topic | Distributed, Parallel, and Cluster Computing Artificial Intelligence Databases |
| url | https://arxiv.org/abs/2304.01468 |