Salvato in:
| Autori principali: | , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2026
|
| Soggetti: | |
| Accesso online: | https://arxiv.org/abs/2604.18043 |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866910148503535616 |
|---|---|
| author | Bader, Jonathan Blumenthal, Edgar Eckardt, Marten Krebs, Justus Witzke, Joel Wysokinska, Xemena Aslan, Haci Ismail Kao, Odej |
| author_facet | Bader, Jonathan Blumenthal, Edgar Eckardt, Marten Krebs, Justus Witzke, Joel Wysokinska, Xemena Aslan, Haci Ismail Kao, Odej |
| contents | In modern distributed systems, efficient resource allocation is a vital aspect to maintain scalability, reduce operational costs, and ensure fast execution even across heterogeneous workloads. Predictive models for resource usage are essential tools for optimizing allocation and preventing system bottlenecks. Predictive memory allocation has asymmetric costs as a key challenge: underallocation causes failures while overallocation wastes memory.
We propose a regression method based on a LightGBM and XGBoost ensemble trained to predict high conditional quantiles. To further account for the high cost of underallocations we add a multiplicative safety factor. With our method we are able to reduce the number of under-allocated jobs from 4.17% to 2.89% and average overallocation from 148% to 44.51% on a real-world dataset of build jobs provided by SAP. We further explore the pareto frontier between optimization for underallocation and for overallocation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2604_18043 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Optimizing Memory Allocation in Distributed Clusters with Predictive Modeling Bader, Jonathan Blumenthal, Edgar Eckardt, Marten Krebs, Justus Witzke, Joel Wysokinska, Xemena Aslan, Haci Ismail Kao, Odej Distributed, Parallel, and Cluster Computing In modern distributed systems, efficient resource allocation is a vital aspect to maintain scalability, reduce operational costs, and ensure fast execution even across heterogeneous workloads. Predictive models for resource usage are essential tools for optimizing allocation and preventing system bottlenecks. Predictive memory allocation has asymmetric costs as a key challenge: underallocation causes failures while overallocation wastes memory. We propose a regression method based on a LightGBM and XGBoost ensemble trained to predict high conditional quantiles. To further account for the high cost of underallocations we add a multiplicative safety factor. With our method we are able to reduce the number of under-allocated jobs from 4.17% to 2.89% and average overallocation from 148% to 44.51% on a real-world dataset of build jobs provided by SAP. We further explore the pareto frontier between optimization for underallocation and for overallocation. |
| title | Optimizing Memory Allocation in Distributed Clusters with Predictive Modeling |
| topic | Distributed, Parallel, and Cluster Computing |
| url | https://arxiv.org/abs/2604.18043 |