Report on the Exascale Meta Scheduler. Deliverable D2.3 of the HORIZON-EUROHPC-JU-2021-COE-01 project MaX (101093374)
Fuente:
Zenodo
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Recurso digital |
| Language: | English |
| Published: |
Zenodo
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866901430993944576 |
|---|---|
| author | Bohm, Ada Baranek, Jakub Garcia, Alberto |
| author_facet | Bohm, Ada Baranek, Jakub Garcia, Alberto |
| contents | <p>This document is about HyperQueue, an advanced meta-scheduling system designed for high-performance computing (HPC) environments. It describes HyperQueue’s innovative approach to task scheduling, which enables efficient execution of complex scientific workflows by dynamically managing computational resources. The document explores the system’s key features, including low-overhead task processing, automatic allocation management, output streaming, and emerging data transfer capabilities. It details how HyperQueue addresses challenges in HPC task scheduling by providing flexible resource allocation, supporting producer-consumer workflows, and offering resilience mechanisms. The text also highlights the system’s current capabilities, its integration with frameworks like AiiDA, and outlines ongoing developments to improve multi-node task handling and data communication in large-scale computational pipelines.<br>The document describes these particular features of HyperQueue and its integration, with particular emphasis on the following key features and integrations.</p> <p><br>Key Features and Capabilities:<br>• Efficient handling of millions of tasks with minimal overhead (~0.1ms per task)<br>• Dynamic resource allocation between multiple compute nodes<br>• Automated management of cluster allocations<br>• Built-in task output streaming to reduce filesystem load<br>• Server resilience through journaling for crash recovery<br>• Support for multi-node tasks and MPI applications<br>• Dynamic task submission through "open jobs" feature<br>• Peer-to-peer data exchange between tasks (in development)</p> <p><br>Applications and Integration:<br>• Successfully integrated with the AiiDA framework for high-throughput scientific<br>workflows<br>• Particularly effective for producer-consumer workflows and non-static DAG computations<br>• Supports complex resource specifications for both tasks and nodes<br>• Enables efficient handling of data dependencies in distributed computations</p> |
| format | Recurso digital |
| id | zenodo_https___doi_org_10_5281_zenodo_19662358 |
| institution | Zenodo |
| language | eng |
| publishDate | 2024 |
| publisher | Zenodo |
| record_format | zenodo |
| spellingShingle | Report on the Exascale Meta Scheduler. Deliverable D2.3 of the HORIZON-EUROHPC-JU-2021-COE-01 project MaX (101093374) Bohm, Ada Baranek, Jakub Garcia, Alberto HyperQueue efficient execution of complex scientific workflows <p>This document is about HyperQueue, an advanced meta-scheduling system designed for high-performance computing (HPC) environments. It describes HyperQueue’s innovative approach to task scheduling, which enables efficient execution of complex scientific workflows by dynamically managing computational resources. The document explores the system’s key features, including low-overhead task processing, automatic allocation management, output streaming, and emerging data transfer capabilities. It details how HyperQueue addresses challenges in HPC task scheduling by providing flexible resource allocation, supporting producer-consumer workflows, and offering resilience mechanisms. The text also highlights the system’s current capabilities, its integration with frameworks like AiiDA, and outlines ongoing developments to improve multi-node task handling and data communication in large-scale computational pipelines.<br>The document describes these particular features of HyperQueue and its integration, with particular emphasis on the following key features and integrations.</p> <p><br>Key Features and Capabilities:<br>• Efficient handling of millions of tasks with minimal overhead (~0.1ms per task)<br>• Dynamic resource allocation between multiple compute nodes<br>• Automated management of cluster allocations<br>• Built-in task output streaming to reduce filesystem load<br>• Server resilience through journaling for crash recovery<br>• Support for multi-node tasks and MPI applications<br>• Dynamic task submission through "open jobs" feature<br>• Peer-to-peer data exchange between tasks (in development)</p> <p><br>Applications and Integration:<br>• Successfully integrated with the AiiDA framework for high-throughput scientific<br>workflows<br>• Particularly effective for producer-consumer workflows and non-static DAG computations<br>• Supports complex resource specifications for both tasks and nodes<br>• Enables efficient handling of data dependencies in distributed computations</p> |
| title | Report on the Exascale Meta Scheduler. Deliverable D2.3 of the HORIZON-EUROHPC-JU-2021-COE-01 project MaX (101093374) |
| topic | HyperQueue efficient execution of complex scientific workflows |
| url | https://doi.org/10.5281/zenodo.19662358 |