DESIGNING SCALABLE ETL PIPELINES FOR MULTI-SOURCE GRAPH DATABASE INGESTION
Fuente:
Zenodo
Gespeichert in:
| 1. Verfasser: | |
|---|---|
| Format: | Recurso digital |
| Veröffentlicht: |
Zenodo
2026
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866901542981861376 |
|---|---|
| author | GUPTA, SUMIT |
| author_facet | GUPTA, SUMIT |
| contents | <p>Graph databases are becoming more and more popular among modern enterprises to represent <br>complex relationships in, for instance, social networks, fraud detection, knowledge <br>management, and recommendation systems. However, these companies still have to deal with <br>the pricy issue of importing data from different sources into the graph structures at a large scale. <br>The present paper proposes a complete set of methods that allow for the creation of ETL <br>pipelines that can handle various sources of data efficiently while they are being moved into <br>Neo4j graph databases at the same time as the quality of the data, the performance, and the <br>reliability of the operations are being sustained. The research sets up a distributed ETL <br>architecture where Apache Spark is used to do parallel processing, Apache Kafka is used for <br>streaming ingestion, and a custom transformation logic is used for the mapping of the schema <br>from the relational, document, and API sources into the graph node and relationship structures. <br>Moreover, the research successively performs an evaluation by conducting experiments with <br>850 million records from five different source systems that are PostgreSQL transactional <br>databases, MongoDB document stores, REST APIs, CSV files, and streaming event sources, <br>and eventually, the optimized pipeline architecture reaches a throughput of 2.4 million records <br>per hour with 99.2% data quality accuracy while the latency for streaming updates remains <br>below 5 seconds. Among the most important design patterns that came out of this research are <br>smart batching strategies that control memory consumption along with throughput, relationship <br>inference algorithms that uncover hidden links from foreign key relationships and semantic <br>patterns, data quality validation frameworks that anticipate schema violations and referential <br>integrity problems before graph insertion, and finally, incremental update techniques that <br>refresh changing source data after synchronization instead of loading it in full every time. <br>Partition-aware processing, connection pooling, and bulk import operations together lead to <br>performance optimization which diminishes the infrastructure cost by 67% in contrast to the <br>naive row-by-row insertion method while enhancing reliability through extensive error <br>handling, dead letter queues, and automated retry logic. The study deals with real-world <br>implementation problems like dealing with schema evolution across sources, managing entity <br>conflicts in multi-source scenarios, fine-tuning Cypher query generation for bulk operations, <br>and tracking pipeline health using custom metrics dashboards. The comparative study shows <br>that processing in batches with 50,000 record chunks gives the best throughput-latency balance, <br>whereas streaming ingestion turns out to be vital for real-time fraud detection and <br>recommendation situations that require sub-minute data freshness. The validated framework <br>paves the way for organizations to turn on graph databases at the enterprise scale, thus enabling <br>relationship-based analytics and traversal queries that relational architectures cannot efficiently </p> |
| format | Recurso digital |
| id | zenodo_https___doi_org_10_5281_zenodo_18736296 |
| institution | Zenodo |
| language | |
| publishDate | 2026 |
| publisher | Zenodo |
| record_format | zenodo |
| spellingShingle | DESIGNING SCALABLE ETL PIPELINES FOR MULTI-SOURCE GRAPH DATABASE INGESTION GUPTA, SUMIT GRAPH DATABASE etl pipelines Neo4j Apache Spark Data Integration Multi Source Ingestion, Scalable Architecture, Real-Time Processing <p>Graph databases are becoming more and more popular among modern enterprises to represent <br>complex relationships in, for instance, social networks, fraud detection, knowledge <br>management, and recommendation systems. However, these companies still have to deal with <br>the pricy issue of importing data from different sources into the graph structures at a large scale. <br>The present paper proposes a complete set of methods that allow for the creation of ETL <br>pipelines that can handle various sources of data efficiently while they are being moved into <br>Neo4j graph databases at the same time as the quality of the data, the performance, and the <br>reliability of the operations are being sustained. The research sets up a distributed ETL <br>architecture where Apache Spark is used to do parallel processing, Apache Kafka is used for <br>streaming ingestion, and a custom transformation logic is used for the mapping of the schema <br>from the relational, document, and API sources into the graph node and relationship structures. <br>Moreover, the research successively performs an evaluation by conducting experiments with <br>850 million records from five different source systems that are PostgreSQL transactional <br>databases, MongoDB document stores, REST APIs, CSV files, and streaming event sources, <br>and eventually, the optimized pipeline architecture reaches a throughput of 2.4 million records <br>per hour with 99.2% data quality accuracy while the latency for streaming updates remains <br>below 5 seconds. Among the most important design patterns that came out of this research are <br>smart batching strategies that control memory consumption along with throughput, relationship <br>inference algorithms that uncover hidden links from foreign key relationships and semantic <br>patterns, data quality validation frameworks that anticipate schema violations and referential <br>integrity problems before graph insertion, and finally, incremental update techniques that <br>refresh changing source data after synchronization instead of loading it in full every time. <br>Partition-aware processing, connection pooling, and bulk import operations together lead to <br>performance optimization which diminishes the infrastructure cost by 67% in contrast to the <br>naive row-by-row insertion method while enhancing reliability through extensive error <br>handling, dead letter queues, and automated retry logic. The study deals with real-world <br>implementation problems like dealing with schema evolution across sources, managing entity <br>conflicts in multi-source scenarios, fine-tuning Cypher query generation for bulk operations, <br>and tracking pipeline health using custom metrics dashboards. The comparative study shows <br>that processing in batches with 50,000 record chunks gives the best throughput-latency balance, <br>whereas streaming ingestion turns out to be vital for real-time fraud detection and <br>recommendation situations that require sub-minute data freshness. The validated framework <br>paves the way for organizations to turn on graph databases at the enterprise scale, thus enabling <br>relationship-based analytics and traversal queries that relational architectures cannot efficiently </p> |
| title | DESIGNING SCALABLE ETL PIPELINES FOR MULTI-SOURCE GRAPH DATABASE INGESTION |
| topic | GRAPH DATABASE etl pipelines Neo4j Apache Spark Data Integration Multi Source Ingestion, Scalable Architecture, Real-Time Processing |
| url | https://doi.org/10.5281/zenodo.18736296 |