Fine-Grained Modeling and Optimization for Intelligent Resource Management in Big Data Processing
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2022
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866913510620921856 |
|---|---|
| author | Lyu, Chenghao Fan, Qi Song, Fei Sinha, Arnab Diao, Yanlei Chen, Wei Ma, Li Feng, Yihui Li, Yaliang Zeng, Kai Zhou, Jingren |
| author_facet | Lyu, Chenghao Fan, Qi Song, Fei Sinha, Arnab Diao, Yanlei Chen, Wei Ma, Li Feng, Yihui Li, Yaliang Zeng, Kai Zhou, Jingren |
| contents | Big data processing at the production scale presents a highly complex environment for resource optimization (RO), a problem crucial for meeting performance goals and budgetary constraints of analytical users. The RO problem is challenging because it involves a set of decisions (the partition count, placement of parallel instances on machines, and resource allocation to each instance), requires multi-objective optimization (MOO), and is compounded by the scale and complexity of big data systems while having to meet stringent time constraints for scheduling. This paper presents a MaxCompute-based integrated system to support multi-objective resource optimization via fine-grained instance-level modeling and optimization. We propose a new architecture that breaks RO into a series of simpler problems, new fine-grained predictive models, and novel optimization methods that exploit these models to make effective instance-level recommendations in a hierarchical MOO framework. Evaluation using production workloads shows that our new RO system could reduce 37-72% latency and 43-78% cost at the same time, compared to the current optimizer and scheduler, while running in 0.02-0.23s. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2207_02026 |
| institution | arXiv |
| publishDate | 2022 |
| record_format | arxiv |
| spellingShingle | Fine-Grained Modeling and Optimization for Intelligent Resource Management in Big Data Processing Lyu, Chenghao Fan, Qi Song, Fei Sinha, Arnab Diao, Yanlei Chen, Wei Ma, Li Feng, Yihui Li, Yaliang Zeng, Kai Zhou, Jingren Databases Distributed, Parallel, and Cluster Computing Big data processing at the production scale presents a highly complex environment for resource optimization (RO), a problem crucial for meeting performance goals and budgetary constraints of analytical users. The RO problem is challenging because it involves a set of decisions (the partition count, placement of parallel instances on machines, and resource allocation to each instance), requires multi-objective optimization (MOO), and is compounded by the scale and complexity of big data systems while having to meet stringent time constraints for scheduling. This paper presents a MaxCompute-based integrated system to support multi-objective resource optimization via fine-grained instance-level modeling and optimization. We propose a new architecture that breaks RO into a series of simpler problems, new fine-grained predictive models, and novel optimization methods that exploit these models to make effective instance-level recommendations in a hierarchical MOO framework. Evaluation using production workloads shows that our new RO system could reduce 37-72% latency and 43-78% cost at the same time, compared to the current optimizer and scheduler, while running in 0.02-0.23s. |
| title | Fine-Grained Modeling and Optimization for Intelligent Resource Management in Big Data Processing |
| topic | Databases Distributed, Parallel, and Cluster Computing |
| url | https://arxiv.org/abs/2207.02026 |