ElastiBench: Scalable Continuous Benchmarking on Cloud FaaS Platforms
Fuente:
arXiv
Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914890454663168 |
|---|---|
| author | Schirmer, Trever Pfandzelter, Tobias Bermbach, David |
| author_facet | Schirmer, Trever Pfandzelter, Tobias Bermbach, David |
| contents | Running microbenchmark suites often and early in the development process enables developers to identify performance issues in their application. Microbenchmark suites of complex applications can comprise hundreds of individual benchmarks and take multiple hours to evaluate meaningfully, making running those benchmarks as part of CI/CD pipelines infeasible. In this paper, we reduce the total execution time of microbenchmark suites by leveraging the massive scalability and elasticity of FaaS (Function-as-a-Service) platforms. While using FaaS enables users to quickly scale up to thousands of parallel function instances to speed up microbenchmarking, the performance variation and low control over the underlying computing resources complicate reliable benchmarking. We demonstrate an architecture for executing microbenchmark suites on cloud FaaS platforms and evaluate it on code changes from an open-source time series database. Our evaluation shows that our prototype can produce reliable results (~95% of performance changes accurately detected) in a quarter of the time (<=15min vs.~4h) and at lower cost ($0.49 vs. ~$1.18) compared to cloud-based virtual machines. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_13528 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | ElastiBench: Scalable Continuous Benchmarking on Cloud FaaS Platforms Schirmer, Trever Pfandzelter, Tobias Bermbach, David Distributed, Parallel, and Cluster Computing Running microbenchmark suites often and early in the development process enables developers to identify performance issues in their application. Microbenchmark suites of complex applications can comprise hundreds of individual benchmarks and take multiple hours to evaluate meaningfully, making running those benchmarks as part of CI/CD pipelines infeasible. In this paper, we reduce the total execution time of microbenchmark suites by leveraging the massive scalability and elasticity of FaaS (Function-as-a-Service) platforms. While using FaaS enables users to quickly scale up to thousands of parallel function instances to speed up microbenchmarking, the performance variation and low control over the underlying computing resources complicate reliable benchmarking. We demonstrate an architecture for executing microbenchmark suites on cloud FaaS platforms and evaluate it on code changes from an open-source time series database. Our evaluation shows that our prototype can produce reliable results (~95% of performance changes accurately detected) in a quarter of the time (<=15min vs.~4h) and at lower cost ($0.49 vs. ~$1.18) compared to cloud-based virtual machines. |
| title | ElastiBench: Scalable Continuous Benchmarking on Cloud FaaS Platforms |
| topic | Distributed, Parallel, and Cluster Computing |
| url | https://arxiv.org/abs/2405.13528 |