Two-dimensional Sparse Parallelism for Large Scale Deep Learning Recommendation Model Training
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913976966709248 |
|---|---|
| author | Zhang, Xin Zhu, Quanyu Xu, Liangbei Huda, Zain Zhou, Wang Fang, Jin van der Staay, Dennis Hu, Yuxi Nie, Jade Yang, Jiyan Yang, Chunzhi |
| author_facet | Zhang, Xin Zhu, Quanyu Xu, Liangbei Huda, Zain Zhou, Wang Fang, Jin van der Staay, Dennis Hu, Yuxi Nie, Jade Yang, Jiyan Yang, Chunzhi |
| contents | The increasing complexity of deep learning recommendation models (DLRM) has led to a growing need for large-scale distributed systems that can efficiently train vast amounts of data. In DLRM, the sparse embedding table is a crucial component for managing sparse categorical features. Typically, these tables in industrial DLRMs contain trillions of parameters, necessitating model parallelism strategies to address memory constraints. However, as training systems expand with massive GPUs, the traditional fully parallelism strategies for embedding table post significant scalability challenges, including imbalance and straggler issues, intensive lookup communication, and heavy embedding activation memory. To overcome these limitations, we propose a novel two-dimensional sparse parallelism approach. Rather than fully sharding tables across all GPUs, our solution introduces data parallelism on top of model parallelism. This enables efficient all-to-all communication and reduces peak memory consumption. Additionally, we have developed the momentum-scaled row-wise AdaGrad algorithm to mitigate performance losses associated with the shift in training paradigms. Our extensive experiments demonstrate that the proposed approach significantly enhances training efficiency while maintaining model performance parity. It achieves nearly linear training speed scaling up to 4K GPUs, setting a new state-of-the-art benchmark for recommendation model training. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2508_03854 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Two-dimensional Sparse Parallelism for Large Scale Deep Learning Recommendation Model Training Zhang, Xin Zhu, Quanyu Xu, Liangbei Huda, Zain Zhou, Wang Fang, Jin van der Staay, Dennis Hu, Yuxi Nie, Jade Yang, Jiyan Yang, Chunzhi Distributed, Parallel, and Cluster Computing Machine Learning The increasing complexity of deep learning recommendation models (DLRM) has led to a growing need for large-scale distributed systems that can efficiently train vast amounts of data. In DLRM, the sparse embedding table is a crucial component for managing sparse categorical features. Typically, these tables in industrial DLRMs contain trillions of parameters, necessitating model parallelism strategies to address memory constraints. However, as training systems expand with massive GPUs, the traditional fully parallelism strategies for embedding table post significant scalability challenges, including imbalance and straggler issues, intensive lookup communication, and heavy embedding activation memory. To overcome these limitations, we propose a novel two-dimensional sparse parallelism approach. Rather than fully sharding tables across all GPUs, our solution introduces data parallelism on top of model parallelism. This enables efficient all-to-all communication and reduces peak memory consumption. Additionally, we have developed the momentum-scaled row-wise AdaGrad algorithm to mitigate performance losses associated with the shift in training paradigms. Our extensive experiments demonstrate that the proposed approach significantly enhances training efficiency while maintaining model performance parity. It achieves nearly linear training speed scaling up to 4K GPUs, setting a new state-of-the-art benchmark for recommendation model training. |
| title | Two-dimensional Sparse Parallelism for Large Scale Deep Learning Recommendation Model Training |
| topic | Distributed, Parallel, and Cluster Computing Machine Learning |
| url | https://arxiv.org/abs/2508.03854 |