DiT-HC: Enabling Efficient Training of Visual Generation Model DiT on HPC-oriented CPU Cluster
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2026
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866914335961382912 |
|---|---|
| author | Zhang, Jinxiao Xu, Yunpu Wu, Xiyong Dong, Runmin Cheng, Shenggan Zhao, Yi Chen, Mengxuan Zheng, Qinrui Liu, Jianting Fu, Haohuan |
| author_facet | Zhang, Jinxiao Xu, Yunpu Wu, Xiyong Dong, Runmin Cheng, Shenggan Zhao, Yi Chen, Mengxuan Zheng, Qinrui Liu, Jianting Fu, Haohuan |
| contents | Generative foundation models have become an important tool for data reconstruction and simulation in scientific computing, showing a tight integration with traditional numerical simulations. At the same time, with the development of new hardware features, such as matrix acceleration units and high-bandwidth memory, CPU-based clusters offer promising opportunities to accelerate and scale such models, facilitating the unification of artificial intelligence and scientific computing. We present DiT-HC, the first system to train and scale the generative model DiT on a next-generation HPC CPU cluster. DiT-HC introduces three key techniques: (1) communication-free tensor parallelism (CFTP) with AutoMem for automated memory-aware dataflow, (2) HCOps, a suite of optimized GEMM and operator kernels leveraging vector and matrix acceleration units, and (3) a custom MPI backend that overlaps computation, communication, and memory movement. Experiments show 8.2 to 87.7 times speedups over native or public CPU libraries and 90.6% weak scaling efficiency on 256 nodes. These results demonstrate the feasibility of large-scale generative model training on CPU clusters and provide new insights for future HPC-AI co-design. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2601_01500 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | DiT-HC: Enabling Efficient Training of Visual Generation Model DiT on HPC-oriented CPU Cluster Zhang, Jinxiao Xu, Yunpu Wu, Xiyong Dong, Runmin Cheng, Shenggan Zhao, Yi Chen, Mengxuan Zheng, Qinrui Liu, Jianting Fu, Haohuan Distributed, Parallel, and Cluster Computing Generative foundation models have become an important tool for data reconstruction and simulation in scientific computing, showing a tight integration with traditional numerical simulations. At the same time, with the development of new hardware features, such as matrix acceleration units and high-bandwidth memory, CPU-based clusters offer promising opportunities to accelerate and scale such models, facilitating the unification of artificial intelligence and scientific computing. We present DiT-HC, the first system to train and scale the generative model DiT on a next-generation HPC CPU cluster. DiT-HC introduces three key techniques: (1) communication-free tensor parallelism (CFTP) with AutoMem for automated memory-aware dataflow, (2) HCOps, a suite of optimized GEMM and operator kernels leveraging vector and matrix acceleration units, and (3) a custom MPI backend that overlaps computation, communication, and memory movement. Experiments show 8.2 to 87.7 times speedups over native or public CPU libraries and 90.6% weak scaling efficiency on 256 nodes. These results demonstrate the feasibility of large-scale generative model training on CPU clusters and provide new insights for future HPC-AI co-design. |
| title | DiT-HC: Enabling Efficient Training of Visual Generation Model DiT on HPC-oriented CPU Cluster |
| topic | Distributed, Parallel, and Cluster Computing |
| url | https://arxiv.org/abs/2601.01500 |