Data Version Management and Machine-Actionable Reproducibility for HPC
Fuente:
arXiv
Saved in:
| Main Authors: | , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914057790947328 |
|---|---|
| author | Knüpfer, Andreas Callow, Timothy J. |
| author_facet | Knüpfer, Andreas Callow, Timothy J. |
| contents | We present a solution for research data version control and machine-actionable reproducibility of data processing for High Performance Computing (HPC) environments and the SLURM batch scheduler. Both aspects are important for research data management and the DataLad tool provides both based on the very prevalent git version control system. However, it is incompatible with HPC batch processing. The presented extension makes it compatible with HPC batch processing with the SLURM scheduler. It solves the fundamental incompatibility so that multiple jobs can be scheduled concurrently on the same data repository. It also avoids inefficient behavior patterns which may emerge on parallel file systems. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2505_06558 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Data Version Management and Machine-Actionable Reproducibility for HPC Knüpfer, Andreas Callow, Timothy J. Distributed, Parallel, and Cluster Computing We present a solution for research data version control and machine-actionable reproducibility of data processing for High Performance Computing (HPC) environments and the SLURM batch scheduler. Both aspects are important for research data management and the DataLad tool provides both based on the very prevalent git version control system. However, it is incompatible with HPC batch processing. The presented extension makes it compatible with HPC batch processing with the SLURM scheduler. It solves the fundamental incompatibility so that multiple jobs can be scheduled concurrently on the same data repository. It also avoids inefficient behavior patterns which may emerge on parallel file systems. |
| title | Data Version Management and Machine-Actionable Reproducibility for HPC |
| topic | Distributed, Parallel, and Cluster Computing |
| url | https://arxiv.org/abs/2505.06558 |