Cephalo: Harnessing Heterogeneous GPU Clusters for Training Transformer Models
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866910697982525440 |
|---|---|
| author | Guo, Runsheng Benson Anand, Utkarsh Chen, Arthur Daudjee, Khuzaima |
| author_facet | Guo, Runsheng Benson Anand, Utkarsh Chen, Arthur Daudjee, Khuzaima |
| contents | Training transformer models requires substantial GPU compute and memory resources. In homogeneous clusters, distributed strategies allocate resources evenly, but this approach is inefficient for heterogeneous clusters, where GPUs differ in power and memory. As high-end GPUs are costly and limited in availability, heterogeneous clusters with diverse GPU types are becoming more common. Existing methods attempt to balance compute across GPUs based on capacity but often underutilize compute due to memory constraints. We present Cephalo, a system that optimizes compute and memory usage by decoupling compute distribution from training state assignment. Cephalo outperforms state-of-the-art methods by achieving significantly higher training throughput while supporting larger models and batch sizes. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_01075 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Cephalo: Harnessing Heterogeneous GPU Clusters for Training Transformer Models Guo, Runsheng Benson Anand, Utkarsh Chen, Arthur Daudjee, Khuzaima Distributed, Parallel, and Cluster Computing Training transformer models requires substantial GPU compute and memory resources. In homogeneous clusters, distributed strategies allocate resources evenly, but this approach is inefficient for heterogeneous clusters, where GPUs differ in power and memory. As high-end GPUs are costly and limited in availability, heterogeneous clusters with diverse GPU types are becoming more common. Existing methods attempt to balance compute across GPUs based on capacity but often underutilize compute due to memory constraints. We present Cephalo, a system that optimizes compute and memory usage by decoupling compute distribution from training state assignment. Cephalo outperforms state-of-the-art methods by achieving significantly higher training throughput while supporting larger models and batch sizes. |
| title | Cephalo: Harnessing Heterogeneous GPU Clusters for Training Transformer Models |
| topic | Distributed, Parallel, and Cluster Computing |
| url | https://arxiv.org/abs/2411.01075 |