A Hybrid Log-Driven and Event-Time Streaming Pipeline: Integrating Kafka Streams with Apache Flink for Real-Time Financial Transaction Processing
Fuente:
Zenodo
Saved in:
| Main Author: | |
|---|---|
| Format: | Recurso digital |
| Published: |
Zenodo
2018
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866901823700336640 |
|---|---|
| author | Jaya Ram Menda |
| author_facet | Jaya Ram Menda |
| contents | <p><span lang="EN-US">Growing demand for real-time, fault-tolerant transaction processing in financial and enterprise environments has accelerated the adoption of distributed stream-processing frameworks such as Kafka Streams and Apache Flink, which together enable continuous ingestion, validation, enrichment, and routing of transactional data with strict correctness guarantees. As organizations confront increasing transaction volumes, complex compliance requirements, and the need for instantaneous insights, traditional batch-oriented or monolithic systems are unable to provide the low-latency responsiveness and horizontal scalability demanded by modern financial workloads. In this context, Kafka Streams offers a lightweight, embedded processing model tightly coupled with Kafka’s durable log, enabling deterministic state management, idempotent updates, and replay-based recovery without external cluster infrastructure. Complementing this, Apache Flink provides a mature event-time processing engine with sophisticated windowing, watermark propagation, and fault-tolerant state handling capable of capturing nuanced temporal patterns and detecting anomalies in continuously evolving datasets. By synthesizing these strengths into a unified architecture, and grounding the design in foundational research on log-centric systems, event-time semantics, and early continuous computation frameworks published prior to September 2017, this article demonstrates how the combined approach supports real-time fraud detection, payment authorization, algorithmic trading pipelines, AML monitoring, and audit-ready ledger transformations. Ultimately, the integrated model delivers strong exactly-once consistency, operational resilience, temporal accuracy, and analytical flexibility qualities that are essential for building next-generation financial platforms under stringent regulatory, latency, and correctness constraints.</span></p> |
| format | Recurso digital |
| id | zenodo_https___doi_org_10_5281_zenodo_18084933 |
| institution | Zenodo |
| language | |
| publishDate | 2018 |
| publisher | Zenodo |
| record_format | zenodo |
| spellingShingle | A Hybrid Log-Driven and Event-Time Streaming Pipeline: Integrating Kafka Streams with Apache Flink for Real-Time Financial Transaction Processing Jaya Ram Menda Real-time processing Kafka Streams Apache Flink event-driven architecture event-time processing transaction pipelines <p><span lang="EN-US">Growing demand for real-time, fault-tolerant transaction processing in financial and enterprise environments has accelerated the adoption of distributed stream-processing frameworks such as Kafka Streams and Apache Flink, which together enable continuous ingestion, validation, enrichment, and routing of transactional data with strict correctness guarantees. As organizations confront increasing transaction volumes, complex compliance requirements, and the need for instantaneous insights, traditional batch-oriented or monolithic systems are unable to provide the low-latency responsiveness and horizontal scalability demanded by modern financial workloads. In this context, Kafka Streams offers a lightweight, embedded processing model tightly coupled with Kafka’s durable log, enabling deterministic state management, idempotent updates, and replay-based recovery without external cluster infrastructure. Complementing this, Apache Flink provides a mature event-time processing engine with sophisticated windowing, watermark propagation, and fault-tolerant state handling capable of capturing nuanced temporal patterns and detecting anomalies in continuously evolving datasets. By synthesizing these strengths into a unified architecture, and grounding the design in foundational research on log-centric systems, event-time semantics, and early continuous computation frameworks published prior to September 2017, this article demonstrates how the combined approach supports real-time fraud detection, payment authorization, algorithmic trading pipelines, AML monitoring, and audit-ready ledger transformations. Ultimately, the integrated model delivers strong exactly-once consistency, operational resilience, temporal accuracy, and analytical flexibility qualities that are essential for building next-generation financial platforms under stringent regulatory, latency, and correctness constraints.</span></p> |
| title | A Hybrid Log-Driven and Event-Time Streaming Pipeline: Integrating Kafka Streams with Apache Flink for Real-Time Financial Transaction Processing |
| topic | Real-time processing Kafka Streams Apache Flink event-driven architecture event-time processing transaction pipelines |
| url | https://doi.org/10.5281/zenodo.18084933 |