Cost-aware scheduling of ML pipelines in heterogeneous cloud environments
Fuente:
Zenodo
Salvato in:
| Autore principale: | |
|---|---|
| Natura: | Recurso digital |
| Lingua: | inglese |
| Pubblicazione: |
Zenodo
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866902097778180096 |
|---|---|
| author | Singh, Ramkinker |
| author_facet | Singh, Ramkinker |
| contents | <p>The rise of complexity and scale in machine learning (ML) workflows and increasing adoption of heterogeneous cloud infrastructures has made cost-effective scheduling of pipelines challenging. Traditional scheduling mechanisms often don't account for variabilities in pricing, efficiency in energy consumption, heterogeneity in resources, or interoperability across clouds, which results in suboptimal costs and inefficiencies in resource utilization. In this paper, we will review the current literature and newer methods that focus on cost-aware scheduling of ML pipelines, in the aforementioned environments. We will focus on intelligent scheduling mechanisms based on reinforcement learning, AI-based scheduling, and optimization, energy aware scheduling policies, and global orchestration. In particular, we will review the recent advances in the use of evolutionary algorithms in scheduling, cloud agnostic scheduling frameworks, and carbon aware scheduling and infrastructure management, to provide a large perspective on how heterogeneous computing environments can be harnessed to increase the performance and cost-effectiveness in ML workflows. We will aim to provide a state-of-the-art overview of methods and approaches that help researchers and practitioners optimize deployment strategies for large-scale ML in multi-cloud and hybrid architecture like exist today.</p> |
| format | Recurso digital |
| id | zenodo_https___doi_org_10_5281_zenodo_17283268 |
| institution | Zenodo |
| language | eng |
| publishDate | 2025 |
| publisher | Zenodo |
| record_format | zenodo |
| spellingShingle | Cost-aware scheduling of ML pipelines in heterogeneous cloud environments Singh, Ramkinker Cost-Aware Scheduling Machine Learning Pipelines Heterogeneous Cloud Resource Optimization <p>The rise of complexity and scale in machine learning (ML) workflows and increasing adoption of heterogeneous cloud infrastructures has made cost-effective scheduling of pipelines challenging. Traditional scheduling mechanisms often don't account for variabilities in pricing, efficiency in energy consumption, heterogeneity in resources, or interoperability across clouds, which results in suboptimal costs and inefficiencies in resource utilization. In this paper, we will review the current literature and newer methods that focus on cost-aware scheduling of ML pipelines, in the aforementioned environments. We will focus on intelligent scheduling mechanisms based on reinforcement learning, AI-based scheduling, and optimization, energy aware scheduling policies, and global orchestration. In particular, we will review the recent advances in the use of evolutionary algorithms in scheduling, cloud agnostic scheduling frameworks, and carbon aware scheduling and infrastructure management, to provide a large perspective on how heterogeneous computing environments can be harnessed to increase the performance and cost-effectiveness in ML workflows. We will aim to provide a state-of-the-art overview of methods and approaches that help researchers and practitioners optimize deployment strategies for large-scale ML in multi-cloud and hybrid architecture like exist today.</p> |
| title | Cost-aware scheduling of ML pipelines in heterogeneous cloud environments |
| topic | Cost-Aware Scheduling Machine Learning Pipelines Heterogeneous Cloud Resource Optimization |
| url | https://doi.org/10.5281/zenodo.17283268 |