Wilkins: HPC In Situ Workflows Made Easy
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866911827968917504 |
|---|---|
| author | Yildiz, Orcun Morozov, Dmitriy Nigmetov, Arnur Nicolae, Bogdan Peterka, Tom |
| author_facet | Yildiz, Orcun Morozov, Dmitriy Nigmetov, Arnur Nicolae, Bogdan Peterka, Tom |
| contents | In situ approaches can accelerate the pace of scientific discoveries by allowing scientists to perform data analysis at simulation time. Current in situ workflow systems, however, face challenges in handling the growing complexity and diverse computational requirements of scientific tasks. In this work, we present Wilkins, an in situ workflow system that is designed for ease-of-use while providing scalable and efficient execution of workflow tasks. Wilkins provides a flexible workflow description interface, employs a high-performance data transport layer based on HDF5, and supports tasks with disparate data rates by providing a flow control mechanism. Wilkins seamlessly couples scientific tasks that already use HDF5, without requiring task code modifications. We demonstrate the above features using both synthetic benchmarks and two science use cases in materials science and cosmology. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_03591 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Wilkins: HPC In Situ Workflows Made Easy Yildiz, Orcun Morozov, Dmitriy Nigmetov, Arnur Nicolae, Bogdan Peterka, Tom Distributed, Parallel, and Cluster Computing In situ approaches can accelerate the pace of scientific discoveries by allowing scientists to perform data analysis at simulation time. Current in situ workflow systems, however, face challenges in handling the growing complexity and diverse computational requirements of scientific tasks. In this work, we present Wilkins, an in situ workflow system that is designed for ease-of-use while providing scalable and efficient execution of workflow tasks. Wilkins provides a flexible workflow description interface, employs a high-performance data transport layer based on HDF5, and supports tasks with disparate data rates by providing a flow control mechanism. Wilkins seamlessly couples scientific tasks that already use HDF5, without requiring task code modifications. We demonstrate the above features using both synthetic benchmarks and two science use cases in materials science and cosmology. |
| title | Wilkins: HPC In Situ Workflows Made Easy |
| topic | Distributed, Parallel, and Cluster Computing |
| url | https://arxiv.org/abs/2404.03591 |