Performance Optimization in Stream Processing Systems: Experiment-Driven Configuration Tuning for Kafka Streams
Fuente:
arXiv
Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912942581088256 |
|---|---|
| author | Chen, David Henning, Sören Matteussi, Kassiano Rabiser, Rick |
| author_facet | Chen, David Henning, Sören Matteussi, Kassiano Rabiser, Rick |
| contents | Configuring stream processing systems for efficient performance, especially in cloud-native deployments, is a challenging and largely manual task. We present an experiment-driven approach for automated configuration optimization that combines three phases: Latin Hypercube Sampling for initial exploration, Simulated Annealing for guided stochastic search, and Hill Climbing for local refinement. The workflow is integrated with the cloud-native Theodolite benchmarking framework, enabling automated experiment orchestration on Kubernetes and early termination of underperforming configurations. In an experimental evaluation with Kafka Streams and a Kubernetes-based cloud testbed, our approach identifies configurations that improve throughput by up to 23% over the default. The results indicate that Latin Hypercube Sampling with early termination and Simulated Annealing are particularly effective in navigating the configuration space, whereas additional fine-tuning via Hill Climbing yields limited benefits. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_04027 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Performance Optimization in Stream Processing Systems: Experiment-Driven Configuration Tuning for Kafka Streams Chen, David Henning, Sören Matteussi, Kassiano Rabiser, Rick Performance Distributed, Parallel, and Cluster Computing Configuring stream processing systems for efficient performance, especially in cloud-native deployments, is a challenging and largely manual task. We present an experiment-driven approach for automated configuration optimization that combines three phases: Latin Hypercube Sampling for initial exploration, Simulated Annealing for guided stochastic search, and Hill Climbing for local refinement. The workflow is integrated with the cloud-native Theodolite benchmarking framework, enabling automated experiment orchestration on Kubernetes and early termination of underperforming configurations. In an experimental evaluation with Kafka Streams and a Kubernetes-based cloud testbed, our approach identifies configurations that improve throughput by up to 23% over the default. The results indicate that Latin Hypercube Sampling with early termination and Simulated Annealing are particularly effective in navigating the configuration space, whereas additional fine-tuning via Hill Climbing yields limited benefits. |
| title | Performance Optimization in Stream Processing Systems: Experiment-Driven Configuration Tuning for Kafka Streams |
| topic | Performance Distributed, Parallel, and Cluster Computing |
| url | https://arxiv.org/abs/2603.04027 |