FreeScale: Distributed Training for Sequence Recommendation Models with Minimal Scaling Cost
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913064497971200 |
|---|---|
| author | Feng, Chenhao Zhang, Haoli Ali-Zade, Shakhzod Zhao, Yanli Luo, Liang Cao, Jennifer Deng, Lisen Chen, Siqiao Zhao, Chenyu Rice, Tristan Johnson, Daniel Si, Min Xu, Tiantu Zhang, Yi Yan, Siqi Zhuge, Chuanhao Ni, Min Xue, Bi Zhang, Qunshu Li, Shen |
| author_facet | Feng, Chenhao Zhang, Haoli Ali-Zade, Shakhzod Zhao, Yanli Luo, Liang Cao, Jennifer Deng, Lisen Chen, Siqiao Zhao, Chenyu Rice, Tristan Johnson, Daniel Si, Min Xu, Tiantu Zhang, Yi Yan, Siqi Zhuge, Chuanhao Ni, Min Xue, Bi Zhang, Qunshu Li, Shen |
| contents | Modern industrial Deep Learning Recommendation Models typically extract user preferences through the analysis of sequential interaction histories, subsequently generating predictions based on these derived interests. The inherent heterogeneity in data characteristics frequently result in substantial under-utilization of computational resources during large-scale training, primarily due to computational bubbles caused by severe stragglers and slow blocking communications. This paper introduces FreeScale, a solution designed to (1) mitigate the straggler problem through meticulously load balanced input samples (2) minimize the blocking communication by overlapping prioritized embedding communications with computations (3) resolve the GPU resource competition during computation and communication overlapping by communicating through SM-Free techniques. Empirical evaluation demonstrates that FreeScale achieves up to 90.3% reduction in computational bubbles when applied to real-world workloads running on 256 H100 GPUs. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_24073 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | FreeScale: Distributed Training for Sequence Recommendation Models with Minimal Scaling Cost Feng, Chenhao Zhang, Haoli Ali-Zade, Shakhzod Zhao, Yanli Luo, Liang Cao, Jennifer Deng, Lisen Chen, Siqiao Zhao, Chenyu Rice, Tristan Johnson, Daniel Si, Min Xu, Tiantu Zhang, Yi Yan, Siqi Zhuge, Chuanhao Ni, Min Xue, Bi Zhang, Qunshu Li, Shen Machine Learning Artificial Intelligence Distributed, Parallel, and Cluster Computing Information Retrieval Modern industrial Deep Learning Recommendation Models typically extract user preferences through the analysis of sequential interaction histories, subsequently generating predictions based on these derived interests. The inherent heterogeneity in data characteristics frequently result in substantial under-utilization of computational resources during large-scale training, primarily due to computational bubbles caused by severe stragglers and slow blocking communications. This paper introduces FreeScale, a solution designed to (1) mitigate the straggler problem through meticulously load balanced input samples (2) minimize the blocking communication by overlapping prioritized embedding communications with computations (3) resolve the GPU resource competition during computation and communication overlapping by communicating through SM-Free techniques. Empirical evaluation demonstrates that FreeScale achieves up to 90.3% reduction in computational bubbles when applied to real-world workloads running on 256 H100 GPUs. |
| title | FreeScale: Distributed Training for Sequence Recommendation Models with Minimal Scaling Cost |
| topic | Machine Learning Artificial Intelligence Distributed, Parallel, and Cluster Computing Information Retrieval |
| url | https://arxiv.org/abs/2604.24073 |