Performance Characterization of Distributed Deep Learning Strategies: A Quantitative Evaluation of DDP, FSDP, and Parameter Server Architectures on GPU Clusters
Fuente:
arXiv
Saved in:
| Main Author: | |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917181763092480 |
|---|---|
| author | Ovi, Md Sultanul Islam |
| author_facet | Ovi, Md Sultanul Islam |
| contents | Efficiently scaling deep neural networks across GPU clusters requires navigating complex trade-offs between computational throughput, memory utilization, and synchronization overhead. This paper presents a unified empirical evaluation of three dominant distributed training paradigms: Distributed Data Parallel (DDP), Fully Sharded Data Parallel (FSDP), and the Parameter Server (PS) architecture. We conduct side-by-side benchmarking on both high-performance (NVIDIA A100) and commodity-class (NVIDIA A10G) clusters to isolate the impact of communication bandwidth and gang-scheduling dependencies. Our results indicate that while DDP achieves a 2-3x speedup in training throughput for standard architectures, FSDP demonstrates a 4-6x reduction in peak memory usage, validating its utility for memory-constrained environments despite higher communication latency. Furthermore, we evaluate the elasticity of the Parameter Server architecture; while Asynchronous PS reduced training time by up to 28% compared to synchronous approaches, it incurred significant accuracy penalties (ranging from 4% to 17%) due to gradient staleness. We also analyze a modified, staleness-mitigating asynchronous protocol, which we found introduced synchronization overheads that negated throughput gains. These findings provide a decision framework for system designers, highlighting that while DDP remains optimal for homogeneous, gang-scheduled clusters, FSDP and PS offer critical alternatives for memory-bound and heterogeneous environments respectively. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_12832 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Performance Characterization of Distributed Deep Learning Strategies: A Quantitative Evaluation of DDP, FSDP, and Parameter Server Architectures on GPU Clusters Ovi, Md Sultanul Islam Distributed, Parallel, and Cluster Computing Efficiently scaling deep neural networks across GPU clusters requires navigating complex trade-offs between computational throughput, memory utilization, and synchronization overhead. This paper presents a unified empirical evaluation of three dominant distributed training paradigms: Distributed Data Parallel (DDP), Fully Sharded Data Parallel (FSDP), and the Parameter Server (PS) architecture. We conduct side-by-side benchmarking on both high-performance (NVIDIA A100) and commodity-class (NVIDIA A10G) clusters to isolate the impact of communication bandwidth and gang-scheduling dependencies. Our results indicate that while DDP achieves a 2-3x speedup in training throughput for standard architectures, FSDP demonstrates a 4-6x reduction in peak memory usage, validating its utility for memory-constrained environments despite higher communication latency. Furthermore, we evaluate the elasticity of the Parameter Server architecture; while Asynchronous PS reduced training time by up to 28% compared to synchronous approaches, it incurred significant accuracy penalties (ranging from 4% to 17%) due to gradient staleness. We also analyze a modified, staleness-mitigating asynchronous protocol, which we found introduced synchronization overheads that negated throughput gains. These findings provide a decision framework for system designers, highlighting that while DDP remains optimal for homogeneous, gang-scheduled clusters, FSDP and PS offer critical alternatives for memory-bound and heterogeneous environments respectively. |
| title | Performance Characterization of Distributed Deep Learning Strategies: A Quantitative Evaluation of DDP, FSDP, and Parameter Server Architectures on GPU Clusters |
| topic | Distributed, Parallel, and Cluster Computing |
| url | https://arxiv.org/abs/2505.12832 |