Technical Note #1: Design Trade-offs and Failure Modes in a Real-Time Market Data Pipeline
Fuente:
Zenodo
Salvato in:
| Autore principale: | |
|---|---|
| Natura: | Recurso digital |
| Lingua: | inglese |
| Pubblicazione: |
Zenodo
2026
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866902197920333824 |
|---|---|
| author | Colella, Charles |
| author_facet | Colella, Charles |
| contents | <p>Technical Note #1 in a series on a distributed real-time market-data system.</p> <p>This note studies the design and operation of a live market-data pipeline that ingests exchange updates, reconstructs per-symbol Level 2 order books, and computes derived streams online. The focus is on how the system remains operational under irregular input, changing load, saturation, recovery events, and bounded compute resources.</p> <p>The analysis covers the pipeline design, the deployed implementation, and a live observation window around the U.S. market open. The case study covers 934 order books and more than 280 million output records, and shows how backlog, delay, explicit loss, and snapshot recovery appear as the system approaches saturation.</p> |
| format | Recurso digital |
| id | zenodo_https___doi_org_10_5281_zenodo_20341315 |
| institution | Zenodo |
| language | eng |
| publishDate | 2026 |
| publisher | Zenodo |
| record_format | zenodo |
| spellingShingle | Technical Note #1: Design Trade-offs and Failure Modes in a Real-Time Market Data Pipeline Colella, Charles real-time systems market data pipelines stream processing order book market microstructure backpressure failure modes recovery mechanisms distributed systems cryptocurrency market data <p>Technical Note #1 in a series on a distributed real-time market-data system.</p> <p>This note studies the design and operation of a live market-data pipeline that ingests exchange updates, reconstructs per-symbol Level 2 order books, and computes derived streams online. The focus is on how the system remains operational under irregular input, changing load, saturation, recovery events, and bounded compute resources.</p> <p>The analysis covers the pipeline design, the deployed implementation, and a live observation window around the U.S. market open. The case study covers 934 order books and more than 280 million output records, and shows how backlog, delay, explicit loss, and snapshot recovery appear as the system approaches saturation.</p> |
| title | Technical Note #1: Design Trade-offs and Failure Modes in a Real-Time Market Data Pipeline |
| topic | real-time systems market data pipelines stream processing order book market microstructure backpressure failure modes recovery mechanisms distributed systems cryptocurrency market data |
| url | https://doi.org/10.5281/zenodo.20341315 |