G-Meta: Distributed Meta Learning in GPU Clusters for Large-Scale Recommender Systems
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866913315394945024 |
|---|---|
| author | Xiao, Youshao Zhao, Shangchun Zhou, Zhenglei Huan, Zhaoxin Ju, Lin Zhang, Xiaolu Wang, Lin Zhou, Jun |
| author_facet | Xiao, Youshao Zhao, Shangchun Zhou, Zhenglei Huan, Zhaoxin Ju, Lin Zhang, Xiaolu Wang, Lin Zhou, Jun |
| contents | Recently, a new paradigm, meta learning, has been widely applied to Deep Learning Recommendation Models (DLRM) and significantly improves statistical performance, especially in cold-start scenarios. However, the existing systems are not tailored for meta learning based DLRM models and have critical problems regarding efficiency in distributed training in the GPU cluster. It is because the conventional deep learning pipeline is not optimized for two task-specific datasets and two update loops in meta learning. This paper provides a high-performance framework for large-scale training for Optimization-based Meta DLRM models over the \textbf{G}PU cluster, namely \textbf{G}-Meta. Firstly, G-Meta utilizes both data parallelism and model parallelism with careful orchestration regarding computation and communication efficiency, to enable high-speed distributed training. Secondly, it proposes a Meta-IO pipeline for efficient data ingestion to alleviate the I/O bottleneck. Various experimental results show that G-Meta achieves notable training speed without loss of statistical performance. Since early 2022, G-Meta has been deployed in Alipay's core advertising and recommender system, shrinking the continuous delivery of models by four times. It also obtains 6.48\% improvement in Conversion Rate (CVR) and 1.06\% increase in CPM (Cost Per Mille) in Alipay's homepage display advertising, with the benefit of larger training samples and tasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2401_04338 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | G-Meta: Distributed Meta Learning in GPU Clusters for Large-Scale Recommender Systems Xiao, Youshao Zhao, Shangchun Zhou, Zhenglei Huan, Zhaoxin Ju, Lin Zhang, Xiaolu Wang, Lin Zhou, Jun Machine Learning Distributed, Parallel, and Cluster Computing Information Retrieval Recently, a new paradigm, meta learning, has been widely applied to Deep Learning Recommendation Models (DLRM) and significantly improves statistical performance, especially in cold-start scenarios. However, the existing systems are not tailored for meta learning based DLRM models and have critical problems regarding efficiency in distributed training in the GPU cluster. It is because the conventional deep learning pipeline is not optimized for two task-specific datasets and two update loops in meta learning. This paper provides a high-performance framework for large-scale training for Optimization-based Meta DLRM models over the \textbf{G}PU cluster, namely \textbf{G}-Meta. Firstly, G-Meta utilizes both data parallelism and model parallelism with careful orchestration regarding computation and communication efficiency, to enable high-speed distributed training. Secondly, it proposes a Meta-IO pipeline for efficient data ingestion to alleviate the I/O bottleneck. Various experimental results show that G-Meta achieves notable training speed without loss of statistical performance. Since early 2022, G-Meta has been deployed in Alipay's core advertising and recommender system, shrinking the continuous delivery of models by four times. It also obtains 6.48\% improvement in Conversion Rate (CVR) and 1.06\% increase in CPM (Cost Per Mille) in Alipay's homepage display advertising, with the benefit of larger training samples and tasks. |
| title | G-Meta: Distributed Meta Learning in GPU Clusters for Large-Scale Recommender Systems |
| topic | Machine Learning Distributed, Parallel, and Cluster Computing Information Retrieval |
| url | https://arxiv.org/abs/2401.04338 |