Predicting the Performance of Scientific Workflow Tasks for Cluster Resource Management: An Overview of the State of the Art
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2025
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866909597071048704 |
|---|---|
| author | Bader, Jonathan West, Kathleen Becker, Soeren Kulagina, Svetlana Lehmann, Fabian Thamsen, Lauritz Meyerhenke, Henning Kao, Odej |
| author_facet | Bader, Jonathan West, Kathleen Becker, Soeren Kulagina, Svetlana Lehmann, Fabian Thamsen, Lauritz Meyerhenke, Henning Kao, Odej |
| contents | Scientific workflow management systems support large-scale data analysis on cluster infrastructures. For this, they interact with resource managers which schedule workflow tasks onto cluster nodes. In addition to workflow task descriptions, resource managers rely on task performance estimates such as main memory consumption and runtime to efficiently manage cluster resources. Such performance estimates should be automated, as user-based task performance estimates are error-prone.
In this book chapter, we describe key characteristics of methods for workflow task runtime and memory prediction, provide an overview and a detailed comparison of state-of-the-art methods from the literature, and discuss how workflow task performance prediction is useful for scheduling, energy-efficient and carbon-aware computing, and cost prediction. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2504_20867 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Predicting the Performance of Scientific Workflow Tasks for Cluster Resource Management: An Overview of the State of the Art Bader, Jonathan West, Kathleen Becker, Soeren Kulagina, Svetlana Lehmann, Fabian Thamsen, Lauritz Meyerhenke, Henning Kao, Odej Distributed, Parallel, and Cluster Computing Scientific workflow management systems support large-scale data analysis on cluster infrastructures. For this, they interact with resource managers which schedule workflow tasks onto cluster nodes. In addition to workflow task descriptions, resource managers rely on task performance estimates such as main memory consumption and runtime to efficiently manage cluster resources. Such performance estimates should be automated, as user-based task performance estimates are error-prone. In this book chapter, we describe key characteristics of methods for workflow task runtime and memory prediction, provide an overview and a detailed comparison of state-of-the-art methods from the literature, and discuss how workflow task performance prediction is useful for scheduling, energy-efficient and carbon-aware computing, and cost prediction. |
| title | Predicting the Performance of Scientific Workflow Tasks for Cluster Resource Management: An Overview of the State of the Art |
| topic | Distributed, Parallel, and Cluster Computing |
| url | https://arxiv.org/abs/2504.20867 |