Saved in:
| Main Authors: | , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2509.07003 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866918137639731200 |
|---|---|
| author | Li, Youjie Wan, Cheng Lin, Zhiqi Zhu, Hongyu Yang, Jiacheng Song, Ziang Di, Xinyi Wu, Jiawei Shu, Huiyao Bao, Wenlei Peng, Yanghua Lin, Haibin Chang, Li-Wen |
| author_facet | Li, Youjie Wan, Cheng Lin, Zhiqi Zhu, Hongyu Yang, Jiacheng Song, Ziang Di, Xinyi Wu, Jiawei Shu, Huiyao Bao, Wenlei Peng, Yanghua Lin, Haibin Chang, Li-Wen |
| contents | Large Language Models (LLMs) have scaled rapidly in size and complexity, requiring increasingly intricate parallelism for distributed training, such as 3D parallelism. This sophistication motivates a shift toward simpler, more debuggable programming paradigm like Single Program Multiple Data (SPMD). However, SPMD in eager execution introduces two key challenges: ensuring consistency with single-device execution and achieving high performance at scale. In this paper, we introduce veScale, an eager-mode training system that fully embraces SPMD paradigm to democratize distributed tensor programming. veScale addresses the prevalent issue of inconsistent results in systems like PyTorch by introducing a novel algorithm of distributed Random Number Generation (RNG) compatible with arbitrary sharded operators. veScale also significantly boosts training performance by reducing PyTorch primitive's overhead and improving communication efficiency. Evaluations show that veScale delivers up to 2.2x speedup over the state-of-the-art training systems, like TorchTitan, and cuts code complexity by 78.4%, while preserving single-device-equivalent results. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2509_07003 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | veScale: Consistent and Efficient Tensor Programming with Eager-Mode SPMD Li, Youjie Wan, Cheng Lin, Zhiqi Zhu, Hongyu Yang, Jiacheng Song, Ziang Di, Xinyi Wu, Jiawei Shu, Huiyao Bao, Wenlei Peng, Yanghua Lin, Haibin Chang, Li-Wen Programming Languages Distributed, Parallel, and Cluster Computing Machine Learning Large Language Models (LLMs) have scaled rapidly in size and complexity, requiring increasingly intricate parallelism for distributed training, such as 3D parallelism. This sophistication motivates a shift toward simpler, more debuggable programming paradigm like Single Program Multiple Data (SPMD). However, SPMD in eager execution introduces two key challenges: ensuring consistency with single-device execution and achieving high performance at scale. In this paper, we introduce veScale, an eager-mode training system that fully embraces SPMD paradigm to democratize distributed tensor programming. veScale addresses the prevalent issue of inconsistent results in systems like PyTorch by introducing a novel algorithm of distributed Random Number Generation (RNG) compatible with arbitrary sharded operators. veScale also significantly boosts training performance by reducing PyTorch primitive's overhead and improving communication efficiency. Evaluations show that veScale delivers up to 2.2x speedup over the state-of-the-art training systems, like TorchTitan, and cuts code complexity by 78.4%, while preserving single-device-equivalent results. |
| title | veScale: Consistent and Efficient Tensor Programming with Eager-Mode SPMD |
| topic | Programming Languages Distributed, Parallel, and Cluster Computing Machine Learning |
| url | https://arxiv.org/abs/2509.07003 |