Designing self-healing ETL pipelines with airflow and databricks
Fuente:
Zenodo
Enregistré dans:
| Auteur principal: | |
|---|---|
| Format: | Recurso digital |
| Langue: | anglais |
| Publié: |
Zenodo
2025
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866901759880855552 |
|---|---|
| author | Veeramachaneni, Jayanth |
| author_facet | Veeramachaneni, Jayanth |
| contents | <p>The growing intricacy and scope of data processing systems have raised the significance of multi-purpose and clever ETL pipelines (Extract, Transform, Load) to the state of deliberations. The data-typical integration has been switched to real-time data integration, which at times makes the self-healing of ETL workflow a requirement. The paper includes the description of the design philosophy, architecture, and the process of practice of self-healing ETL pipelines creation with the help of Apache Airflow and Databricks. It provides a clue of how the ETL systems are going to transform themselves in the recent past to be event-driven and AI-enhanced pipes in the cloud and serverless worlds. It is concerned with alerts in a fault, automated recovery, generative AI-assisted, and distributed architecture-assisted pipeline adaptivity. The review also includes modern techniques and emerging technologies, and this has helped ETL systems to automatically detect, troubleshoot, and remediate failure and the resultant effect is low downtime and the result load. </p> |
| format | Recurso digital |
| id | zenodo_https___doi_org_10_5281_zenodo_18337295 |
| institution | Zenodo |
| language | eng |
| publishDate | 2025 |
| publisher | Zenodo |
| record_format | zenodo |
| spellingShingle | Designing self-healing ETL pipelines with airflow and databricks Veeramachaneni, Jayanth Self-Healing ETL Airflow Databricks Pipeline Automation <p>The growing intricacy and scope of data processing systems have raised the significance of multi-purpose and clever ETL pipelines (Extract, Transform, Load) to the state of deliberations. The data-typical integration has been switched to real-time data integration, which at times makes the self-healing of ETL workflow a requirement. The paper includes the description of the design philosophy, architecture, and the process of practice of self-healing ETL pipelines creation with the help of Apache Airflow and Databricks. It provides a clue of how the ETL systems are going to transform themselves in the recent past to be event-driven and AI-enhanced pipes in the cloud and serverless worlds. It is concerned with alerts in a fault, automated recovery, generative AI-assisted, and distributed architecture-assisted pipeline adaptivity. The review also includes modern techniques and emerging technologies, and this has helped ETL systems to automatically detect, troubleshoot, and remediate failure and the resultant effect is low downtime and the result load. </p> |
| title | Designing self-healing ETL pipelines with airflow and databricks |
| topic | Self-Healing ETL Airflow Databricks Pipeline Automation |
| url | https://doi.org/10.5281/zenodo.18337295 |