Data Dams: A Novel Framework for Regulating and Managing Data Flow in Large-Scale Systems
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866917913991053312 |
|---|---|
| author | Bouke, Mohamed Aly Abdullah, Azizol Cengiz, Korhan Ivković, Nikola Mihaljević, Ivan Mohamud, Mudathir Ahmed Kowrina, Ahmed |
| author_facet | Bouke, Mohamed Aly Abdullah, Azizol Cengiz, Korhan Ivković, Nikola Mihaljević, Ivan Mohamud, Mudathir Ahmed Kowrina, Ahmed |
| contents | In the era of big data, managing dynamic data flows efficiently is crucial as traditional storage models struggle with real-time regulation and risk overflow. This paper introduces Data Dams, a novel framework designed to optimize data inflow, storage, and outflow by dynamically adjusting flow rates to prevent congestion while maximizing resource utilization. Inspired by physical dam mechanisms, the framework employs intelligent sluice controls and predictive analytics to regulate data flow based on system conditions such as bandwidth availability, processing capacity, and security constraints. Simulation results demonstrate that the Data Dam significantly reduces average storage levels (371.68 vs. 426.27 units) and increases total outflow (7999.99 vs. 7748.76 units) compared to static baseline models. By ensuring stable and adaptive outflow rates under fluctuating data loads, this approach enhances system efficiency, mitigates overflow risks, and outperforms existing static flow control strategies. The proposed framework presents a scalable solution for dynamic data management in large-scale distributed systems, paving the way for more resilient and efficient real-time processing architectures. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_03218 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Data Dams: A Novel Framework for Regulating and Managing Data Flow in Large-Scale Systems Bouke, Mohamed Aly Abdullah, Azizol Cengiz, Korhan Ivković, Nikola Mihaljević, Ivan Mohamud, Mudathir Ahmed Kowrina, Ahmed Information Retrieval Databases Distributed, Parallel, and Cluster Computing In the era of big data, managing dynamic data flows efficiently is crucial as traditional storage models struggle with real-time regulation and risk overflow. This paper introduces Data Dams, a novel framework designed to optimize data inflow, storage, and outflow by dynamically adjusting flow rates to prevent congestion while maximizing resource utilization. Inspired by physical dam mechanisms, the framework employs intelligent sluice controls and predictive analytics to regulate data flow based on system conditions such as bandwidth availability, processing capacity, and security constraints. Simulation results demonstrate that the Data Dam significantly reduces average storage levels (371.68 vs. 426.27 units) and increases total outflow (7999.99 vs. 7748.76 units) compared to static baseline models. By ensuring stable and adaptive outflow rates under fluctuating data loads, this approach enhances system efficiency, mitigates overflow risks, and outperforms existing static flow control strategies. The proposed framework presents a scalable solution for dynamic data management in large-scale distributed systems, paving the way for more resilient and efficient real-time processing architectures. |
| title | Data Dams: A Novel Framework for Regulating and Managing Data Flow in Large-Scale Systems |
| topic | Information Retrieval Databases Distributed, Parallel, and Cluster Computing |
| url | https://arxiv.org/abs/2502.03218 |