Mathematical Foundations of Modeling ETL Process Chains
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2026
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912992537346048 |
|---|---|
| author | Maier, Levin Schulze, Lucas Lilow, Robert Hahn, Lukas Krasowski, Nikola Barth, Arnulf Gaebel, Sebastian Güran, Ferdi Hanau, Oliver Wagner, Giovanni Borgmann, Falk Arenz, Oleg Peters, Jan |
| author_facet | Maier, Levin Schulze, Lucas Lilow, Robert Hahn, Lukas Krasowski, Nikola Barth, Arnulf Gaebel, Sebastian Güran, Ferdi Hanau, Oliver Wagner, Giovanni Borgmann, Falk Arenz, Oleg Peters, Jan |
| contents | Extract-Transform-Load (ETL) processes are core components of modern data processing infrastructures. The throughput of processed data records can be adjusted by changing the amount of allocated resources, i.e.~the number of parallel processing threads for each of the three ETL phases, but also depends on stochastic variations in the per-record processing times. In chains of multiple consecutive ETL processes, the relation between allocated resources and overall throughput is further complicated, for example by the occurrence of bottlenecks affecting all subsequent ETL processes. We develop a mathematical model of ETL process chains that is accurate at the level of time-aggregated throughput and suitable for efficient simulation. The process chain is represented as a controlled discrete-time Markov process on a directed acyclic graph whose edges are individual ETL processes. We model the mean throughput as a bounded, monotone function of the number of parallel threads, to capture the diminishing benefit of allocating more threads. We furthermore introduce a Flow Balance postulate linking number of threads, mean throughput, and mean processing time. The stochastic processing times are then modeled by non-negative heavy-tailed distributions around the mean processing time. This framework provides a principled simulator for ETL networks and a foundation for learning- and control-based resource allocation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2603_29877 |
| institution | arXiv |
| publishDate | 2026 |
| record_format | arxiv |
| spellingShingle | Mathematical Foundations of Modeling ETL Process Chains Maier, Levin Schulze, Lucas Lilow, Robert Hahn, Lukas Krasowski, Nikola Barth, Arnulf Gaebel, Sebastian Güran, Ferdi Hanau, Oliver Wagner, Giovanni Borgmann, Falk Arenz, Oleg Peters, Jan Distributed, Parallel, and Cluster Computing Databases Machine Learning Optimization and Control Extract-Transform-Load (ETL) processes are core components of modern data processing infrastructures. The throughput of processed data records can be adjusted by changing the amount of allocated resources, i.e.~the number of parallel processing threads for each of the three ETL phases, but also depends on stochastic variations in the per-record processing times. In chains of multiple consecutive ETL processes, the relation between allocated resources and overall throughput is further complicated, for example by the occurrence of bottlenecks affecting all subsequent ETL processes. We develop a mathematical model of ETL process chains that is accurate at the level of time-aggregated throughput and suitable for efficient simulation. The process chain is represented as a controlled discrete-time Markov process on a directed acyclic graph whose edges are individual ETL processes. We model the mean throughput as a bounded, monotone function of the number of parallel threads, to capture the diminishing benefit of allocating more threads. We furthermore introduce a Flow Balance postulate linking number of threads, mean throughput, and mean processing time. The stochastic processing times are then modeled by non-negative heavy-tailed distributions around the mean processing time. This framework provides a principled simulator for ETL networks and a foundation for learning- and control-based resource allocation. |
| title | Mathematical Foundations of Modeling ETL Process Chains |
| topic | Distributed, Parallel, and Cluster Computing Databases Machine Learning Optimization and Control |
| url | https://arxiv.org/abs/2603.29877 |