Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2503.21033 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866913761003044864 |
|---|---|
| author | Mileski, Dimitar Petrovski, Nikola Gusev, Marjan |
| author_facet | Mileski, Dimitar Petrovski, Nikola Gusev, Marjan |
| contents | Training large language models requires extensive processing, made possible by many high-performance computing resources. This study compares multi-node and multi-GPU environments for training large language models of electrocardiograms. It provides a detailed mapping of current frameworks for distributed deep learning in multinode and multi-GPU settings, including Horovod from Uber, DeepSpeed from Microsoft, and the built-in distributed capabilities of PyTorch and TensorFlow. We compare various multi-GPU setups for different dataset configurations, utilizing multiple HPC nodes independently and focusing on scalability, speedup, efficiency, and overhead. The analysis leverages HPC infrastructure with SLURM, Apptainer (Singularity) containers, CUDA, PyTorch, and shell scripts to support training workflows and automation. We achieved a sub-linear speedup when scaling the number of GPUs, with values of 1.6x for two and 1.9x for four. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2503_21033 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Scalability Evaluation of HPC Multi-GPU Training for ECG-based LLMs Mileski, Dimitar Petrovski, Nikola Gusev, Marjan Distributed, Parallel, and Cluster Computing C.4; I.6 Training large language models requires extensive processing, made possible by many high-performance computing resources. This study compares multi-node and multi-GPU environments for training large language models of electrocardiograms. It provides a detailed mapping of current frameworks for distributed deep learning in multinode and multi-GPU settings, including Horovod from Uber, DeepSpeed from Microsoft, and the built-in distributed capabilities of PyTorch and TensorFlow. We compare various multi-GPU setups for different dataset configurations, utilizing multiple HPC nodes independently and focusing on scalability, speedup, efficiency, and overhead. The analysis leverages HPC infrastructure with SLURM, Apptainer (Singularity) containers, CUDA, PyTorch, and shell scripts to support training workflows and automation. We achieved a sub-linear speedup when scaling the number of GPUs, with values of 1.6x for two and 1.9x for four. |
| title | Scalability Evaluation of HPC Multi-GPU Training for ECG-based LLMs |
| topic | Distributed, Parallel, and Cluster Computing C.4; I.6 |
| url | https://arxiv.org/abs/2503.21033 |