| _version_ | 1866901471442763776 |
|---|---|
| author | Menzies, Ignatius |
| author_facet | Menzies, Ignatius |
| contents | Continuous Integration (CI) is essential in modern software development, enabling frequent code integration, automated testing, and rapid feedback. In eResearch, CI enhances reproducibility and efficiency, but High Performance Computing (HPC) environments pose unique challenges, such as hardware diversity, job scheduling, resource management requirements, and strict administrative policies. We developed a system that integrates CI with HPC to validate changes to bioinformatic workflows. Our system uses GitHub Actions, runners hosted on Google Cloud Platform, and the National Computational Infrastructure s Gadi supercomputer. Any committed changes to workflows trigger a job on Gadi, running the amended workflow with a sample dataset and a suite of tests. This approach regularly integrates code changes, reducing the likelihood of errors in production workflow runs. The tests run on the same hardware as the production workflows, ensuring consistency. While frequently running small validation jobs consumes resources, it significantly reduces errors during much larger production jobs. This system leverages existing HPC access mechanisms, requiring less internal integration with HPC administration compared to other frameworks and tools like Jacamar CI and Jenkins plugins. By leveraging CI, we can significantly enhance the quality, productivity, and resource efficiency of eResearch on HPC infrastructures. |
| format | Recurso digital |
| id | zenodo_https___doi_org_10_5281_zenodo_15293471 |
| institution | Zenodo |
| language | |
| publishDate | 2024 |
| publisher | Zenodo |
| record_format | zenodo |
| spellingShingle | Leveraging Continuous Integration for enhanced eResearch on High Performance Computing Clusters. Menzies, Ignatius Training Skills Analytics Tools Software Infrastructure Continuous Integration (CI) is essential in modern software development, enabling frequent code integration, automated testing, and rapid feedback. In eResearch, CI enhances reproducibility and efficiency, but High Performance Computing (HPC) environments pose unique challenges, such as hardware diversity, job scheduling, resource management requirements, and strict administrative policies. We developed a system that integrates CI with HPC to validate changes to bioinformatic workflows. Our system uses GitHub Actions, runners hosted on Google Cloud Platform, and the National Computational Infrastructure s Gadi supercomputer. Any committed changes to workflows trigger a job on Gadi, running the amended workflow with a sample dataset and a suite of tests. This approach regularly integrates code changes, reducing the likelihood of errors in production workflow runs. The tests run on the same hardware as the production workflows, ensuring consistency. While frequently running small validation jobs consumes resources, it significantly reduces errors during much larger production jobs. This system leverages existing HPC access mechanisms, requiring less internal integration with HPC administration compared to other frameworks and tools like Jacamar CI and Jenkins plugins. By leveraging CI, we can significantly enhance the quality, productivity, and resource efficiency of eResearch on HPC infrastructures. |
| title | Leveraging Continuous Integration for enhanced eResearch on High Performance Computing Clusters. |
| topic | Training Skills Analytics Tools Software Infrastructure |
| url | https://doi.org/10.5281/zenodo.15293471 |