ASA -- The Adaptive Scheduling Algorithm
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911760212033536 |
|---|---|
| author | Souza, Abel Pelckmans, Kristiaan Ghoshal, Devarshi Ramakrishnan, Lavanya Tordsson, Johan |
| author_facet | Souza, Abel Pelckmans, Kristiaan Ghoshal, Devarshi Ramakrishnan, Lavanya Tordsson, Johan |
| contents | In High Performance Computing (HPC) infrastructures, the control of resources by batch systems can lead to prolonged queue waiting times and adverse effects on the overall execution times of applications, particularly in data-intensive and low-latency workflows where efficient processing hinges on resource planning and timely allocation. Allocating the maximum capacity upfront ensures the fastest execution but results in spare and idle resources, extended queue waits, and costly usage. Conversely, dynamic allocation based on workflow stage requirements optimizes resource usage but may negatively impact the total workflow makespan. To address these issues, we introduce ASA, the Adaptive Scheduling Algorithm. ASA is a novel, convergence-proven scheduling technique that minimizes jobs inter-stage waiting times by estimating the queue waiting times to proactively submit resource change requests ahead of time. It strikes a balance between exploration and exploitation, considering both learning (waiting times) and applying learnt insights. Real-world experiments over two supercomputers centers with scientific workflows demonstrate ASA's effectiveness, achieving near-optimal resource utilization and accuracy, with up to 10% and 2% reductions in average workflow queue waiting times and makespan, respectively. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2401_09733 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | ASA -- The Adaptive Scheduling Algorithm Souza, Abel Pelckmans, Kristiaan Ghoshal, Devarshi Ramakrishnan, Lavanya Tordsson, Johan Distributed, Parallel, and Cluster Computing In High Performance Computing (HPC) infrastructures, the control of resources by batch systems can lead to prolonged queue waiting times and adverse effects on the overall execution times of applications, particularly in data-intensive and low-latency workflows where efficient processing hinges on resource planning and timely allocation. Allocating the maximum capacity upfront ensures the fastest execution but results in spare and idle resources, extended queue waits, and costly usage. Conversely, dynamic allocation based on workflow stage requirements optimizes resource usage but may negatively impact the total workflow makespan. To address these issues, we introduce ASA, the Adaptive Scheduling Algorithm. ASA is a novel, convergence-proven scheduling technique that minimizes jobs inter-stage waiting times by estimating the queue waiting times to proactively submit resource change requests ahead of time. It strikes a balance between exploration and exploitation, considering both learning (waiting times) and applying learnt insights. Real-world experiments over two supercomputers centers with scientific workflows demonstrate ASA's effectiveness, achieving near-optimal resource utilization and accuracy, with up to 10% and 2% reductions in average workflow queue waiting times and makespan, respectively. |
| title | ASA -- The Adaptive Scheduling Algorithm |
| topic | Distributed, Parallel, and Cluster Computing |
| url | https://arxiv.org/abs/2401.09733 |