1.5 Million Messages Per Second on 3 Machines: Benchmarking and Latency Optimization of Apache Pulsar at Enterprise Scale
Fuente:
arXiv
Gespeichert in:
| 1. Verfasser: | |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2026
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866910088102412288 |
|---|---|
| author | Mukkolakkal, Muhamed Ramees Cheriya |
| author_facet | Mukkolakkal, Muhamed Ramees Cheriya |
| contents | This paper presents two independent contributions for Apache Pulsar practitioners. First, we validate 1,499,947 msg/s at 3.88 ms median publish latency on just three bare-metal Kubernetes nodes running Pulsar 4.0.8 with Java 21 and ZGC Generational garbage collection, and project a hardware-driven path to 15 million msg/s on 15 machines using five independent clusters with key-based partition routing. Hardware selection -- specifically dedicated NVMe journals achieving 0.02 ms fdatasync and 25 Gbps network interfaces -- is the primary determinant of throughput ceiling, not compute or software tuning. Second, we trace the complete latency optimization journey from 213 ms GC spikes and 13-18 ms median publish latency in production to 3.88 ms through Java Flight Recorder guided root cause analysis. Three independent root causes are identified and resolved: G1GC pauses eliminated by switching to ZGC Generational; journal fdatasync latency reduced from 5.1 ms to 0.02 ms through NVMe journal dedication; and a previously undocumented Linux kernel page cache writeback interaction inside BookKeeper's ForceWriteThread that degrades fdatasync from under 1 ms to 15-22 ms even across physically separate NVMe drives sharing the kernel block layer. This finding is undocumented in official Apache Pulsar and BookKeeper documentation and is relevant to all Pulsar operators experiencing unexplained P99.9 latency spikes. The combined optimizations achieve a 4.7x latency improvement at 50x higher throughput. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_29113 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | 1.5 Million Messages Per Second on 3 Machines: Benchmarking and Latency Optimization of Apache Pulsar at Enterprise Scale Mukkolakkal, Muhamed Ramees Cheriya Distributed, Parallel, and Cluster Computing Networking and Internet Architecture This paper presents two independent contributions for Apache Pulsar practitioners. First, we validate 1,499,947 msg/s at 3.88 ms median publish latency on just three bare-metal Kubernetes nodes running Pulsar 4.0.8 with Java 21 and ZGC Generational garbage collection, and project a hardware-driven path to 15 million msg/s on 15 machines using five independent clusters with key-based partition routing. Hardware selection -- specifically dedicated NVMe journals achieving 0.02 ms fdatasync and 25 Gbps network interfaces -- is the primary determinant of throughput ceiling, not compute or software tuning. Second, we trace the complete latency optimization journey from 213 ms GC spikes and 13-18 ms median publish latency in production to 3.88 ms through Java Flight Recorder guided root cause analysis. Three independent root causes are identified and resolved: G1GC pauses eliminated by switching to ZGC Generational; journal fdatasync latency reduced from 5.1 ms to 0.02 ms through NVMe journal dedication; and a previously undocumented Linux kernel page cache writeback interaction inside BookKeeper's ForceWriteThread that degrades fdatasync from under 1 ms to 15-22 ms even across physically separate NVMe drives sharing the kernel block layer. This finding is undocumented in official Apache Pulsar and BookKeeper documentation and is relevant to all Pulsar operators experiencing unexplained P99.9 latency spikes. The combined optimizations achieve a 4.7x latency improvement at 50x higher throughput. |
| title | 1.5 Million Messages Per Second on 3 Machines: Benchmarking and Latency Optimization of Apache Pulsar at Enterprise Scale |
| topic | Distributed, Parallel, and Cluster Computing Networking and Internet Architecture |
| url | https://arxiv.org/abs/2603.29113 |