Investigating the Impact of Isolation on Synchronized Benchmarks
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866915600179134464 |
|---|---|
| author | Japke, Nils Hamdan, Furat Baumann, Diana Bermbach, David |
| author_facet | Japke, Nils Hamdan, Furat Baumann, Diana Bermbach, David |
| contents | Benchmarking in cloud environments suffers from performance variability from multi-tenant resource contention. Duet benchmarking mitigates this by running two workload versions concurrently on the same VM, exposing them to identical external interference. However, intra-VM contention between synchronized workloads necessitates additional isolation mechanisms.
This work evaluates three such strategies: cgroups and CPU pinning, Docker containers, and Firecracker MicroVMs. We compare all strategies with an unisolated baseline experiment, by running benchmarks with a duet setup alongside a noise generator. This noise generator "steals" compute resources to degrade performance measurements.
All experiments showed different latency distributions while under the effects of noise generation, but results show that process isolation generally lowered false positives, except for our experiments with Docker containers. Even though Docker containers rely internally on cgroups and CPU pinning, they were more susceptible to performance degradation due to noise influence. Therefore, we recommend to use process isolation for synchronized workloads, with the exception of Docker containers. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2511_03533 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Investigating the Impact of Isolation on Synchronized Benchmarks Japke, Nils Hamdan, Furat Baumann, Diana Bermbach, David Distributed, Parallel, and Cluster Computing Software Engineering Benchmarking in cloud environments suffers from performance variability from multi-tenant resource contention. Duet benchmarking mitigates this by running two workload versions concurrently on the same VM, exposing them to identical external interference. However, intra-VM contention between synchronized workloads necessitates additional isolation mechanisms. This work evaluates three such strategies: cgroups and CPU pinning, Docker containers, and Firecracker MicroVMs. We compare all strategies with an unisolated baseline experiment, by running benchmarks with a duet setup alongside a noise generator. This noise generator "steals" compute resources to degrade performance measurements. All experiments showed different latency distributions while under the effects of noise generation, but results show that process isolation generally lowered false positives, except for our experiments with Docker containers. Even though Docker containers rely internally on cgroups and CPU pinning, they were more susceptible to performance degradation due to noise influence. Therefore, we recommend to use process isolation for synchronized workloads, with the exception of Docker containers. |
| title | Investigating the Impact of Isolation on Synchronized Benchmarks |
| topic | Distributed, Parallel, and Cluster Computing Software Engineering |
| url | https://arxiv.org/abs/2511.03533 |