FastPersist: Accelerating Model Checkpointing in Deep Learning
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866910495354650624 |
|---|---|
| author | Wang, Guanhua Ruwase, Olatunji Xie, Bing He, Yuxiong |
| author_facet | Wang, Guanhua Ruwase, Olatunji Xie, Bing He, Yuxiong |
| contents | Model checkpoints are critical Deep Learning (DL) artifacts that enable fault tolerance for training and downstream applications, such as inference. However, writing checkpoints to persistent storage, and other I/O aspects of DL training, are mostly ignored by compute-focused optimization efforts for faster training of rapidly growing models and datasets. Towards addressing this imbalance, we propose FastPersist to accelerate checkpoint creation in DL training. FastPersist combines three novel techniques: (i) NVMe optimizations for faster checkpoint writes to SSDs, (ii) efficient write parallelism using the available SSDs in training environments, and (iii) overlapping checkpointing with independent training computations. Our evaluation using real world dense and sparse DL models shows that FastPersist creates checkpoints in persistent storage up to 116x faster than baseline, and enables per-iteration checkpointing with negligible overhead. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_13768 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | FastPersist: Accelerating Model Checkpointing in Deep Learning Wang, Guanhua Ruwase, Olatunji Xie, Bing He, Yuxiong Distributed, Parallel, and Cluster Computing Artificial Intelligence Machine Learning Performance Model checkpoints are critical Deep Learning (DL) artifacts that enable fault tolerance for training and downstream applications, such as inference. However, writing checkpoints to persistent storage, and other I/O aspects of DL training, are mostly ignored by compute-focused optimization efforts for faster training of rapidly growing models and datasets. Towards addressing this imbalance, we propose FastPersist to accelerate checkpoint creation in DL training. FastPersist combines three novel techniques: (i) NVMe optimizations for faster checkpoint writes to SSDs, (ii) efficient write parallelism using the available SSDs in training environments, and (iii) overlapping checkpointing with independent training computations. Our evaluation using real world dense and sparse DL models shows that FastPersist creates checkpoints in persistent storage up to 116x faster than baseline, and enables per-iteration checkpointing with negligible overhead. |
| title | FastPersist: Accelerating Model Checkpointing in Deep Learning |
| topic | Distributed, Parallel, and Cluster Computing Artificial Intelligence Machine Learning Performance |
| url | https://arxiv.org/abs/2406.13768 |