A Theoretical Framework for AI-driven data quality monitoring in high-volume data environments
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866916433593630720 |
|---|---|
| author | Bangad, Nikhil Jayaram, Vivekananda Krishnappa, Manjunatha Sughaturu Banarse, Amey Ram Bidkar, Darshan Mohan Nagpal, Akshay Parlapalli, Vidyasagar |
| author_facet | Bangad, Nikhil Jayaram, Vivekananda Krishnappa, Manjunatha Sughaturu Banarse, Amey Ram Bidkar, Darshan Mohan Nagpal, Akshay Parlapalli, Vidyasagar |
| contents | This paper presents a theoretical framework for an AI-driven data quality monitoring system designed to address the challenges of maintaining data quality in high-volume environments. We examine the limitations of traditional methods in managing the scale, velocity, and variety of big data and propose a conceptual approach leveraging advanced machine learning techniques. Our framework outlines a system architecture that incorporates anomaly detection, classification, and predictive analytics for real-time, scalable data quality management. Key components include an intelligent data ingestion layer, adaptive preprocessing mechanisms, context-aware feature extraction, and AI-based quality assessment modules. A continuous learning paradigm is central to our framework, ensuring adaptability to evolving data patterns and quality requirements. We also address implications for scalability, privacy, and integration within existing data ecosystems. While practical results are not provided, it lays a robust theoretical foundation for future research and implementations, advancing data quality management and encouraging the exploration of AI-driven solutions in dynamic environments. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2410_08576 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | A Theoretical Framework for AI-driven data quality monitoring in high-volume data environments Bangad, Nikhil Jayaram, Vivekananda Krishnappa, Manjunatha Sughaturu Banarse, Amey Ram Bidkar, Darshan Mohan Nagpal, Akshay Parlapalli, Vidyasagar Artificial Intelligence This paper presents a theoretical framework for an AI-driven data quality monitoring system designed to address the challenges of maintaining data quality in high-volume environments. We examine the limitations of traditional methods in managing the scale, velocity, and variety of big data and propose a conceptual approach leveraging advanced machine learning techniques. Our framework outlines a system architecture that incorporates anomaly detection, classification, and predictive analytics for real-time, scalable data quality management. Key components include an intelligent data ingestion layer, adaptive preprocessing mechanisms, context-aware feature extraction, and AI-based quality assessment modules. A continuous learning paradigm is central to our framework, ensuring adaptability to evolving data patterns and quality requirements. We also address implications for scalability, privacy, and integration within existing data ecosystems. While practical results are not provided, it lays a robust theoretical foundation for future research and implementations, advancing data quality management and encouraging the exploration of AI-driven solutions in dynamic environments. |
| title | A Theoretical Framework for AI-driven data quality monitoring in high-volume data environments |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2410.08576 |