Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2512.05559 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866917127785545728 |
|---|---|
| author | Saini, Devender Jain, Bhavika Ujjwal, Nitish Sommer, Philip Mbanga, Dan Romuald Mehta, Dhagash |
| author_facet | Saini, Devender Jain, Bhavika Ujjwal, Nitish Sommer, Philip Mbanga, Dan Romuald Mehta, Dhagash |
| contents | In regulated domains such as finance, the integrity and governance of data pipelines are critical - yet existing systems treat data quality control (QC) as an isolated preprocessing step rather than a first-class system component. We present a unified AI-driven Data QC and DataOps Management framework that embeds rule-based, statistical, and AI-based QC methods into a continuous, governed layer spanning ingestion, model pipelines, and downstream applications. Our architecture integrates open-source tools with custom modules for profiling, audit logging, breach handling, configuration-driven policies, and dynamic remediation. We demonstrate deployment in a production-grade financial setup: handling streaming and tabular data across multiple asset classes and transaction streams, with configurable thresholds, cloud-native storage interfaces, and automated alerts. We show empirical gains in anomaly detection recall, reduction of manual remediation effort, and improved auditability and traceability in high-throughput data workflows. By treating QC as a system concern rather than an afterthought, our framework provides a foundation for trustworthy, scalable, and compliant AI pipelines in regulated environments. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2512_05559 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | A Unified AI System For Data Quality Control and DataOps Management in Regulated Environments Saini, Devender Jain, Bhavika Ujjwal, Nitish Sommer, Philip Mbanga, Dan Romuald Mehta, Dhagash Computational Finance In regulated domains such as finance, the integrity and governance of data pipelines are critical - yet existing systems treat data quality control (QC) as an isolated preprocessing step rather than a first-class system component. We present a unified AI-driven Data QC and DataOps Management framework that embeds rule-based, statistical, and AI-based QC methods into a continuous, governed layer spanning ingestion, model pipelines, and downstream applications. Our architecture integrates open-source tools with custom modules for profiling, audit logging, breach handling, configuration-driven policies, and dynamic remediation. We demonstrate deployment in a production-grade financial setup: handling streaming and tabular data across multiple asset classes and transaction streams, with configurable thresholds, cloud-native storage interfaces, and automated alerts. We show empirical gains in anomaly detection recall, reduction of manual remediation effort, and improved auditability and traceability in high-throughput data workflows. By treating QC as a system concern rather than an afterthought, our framework provides a foundation for trustworthy, scalable, and compliant AI pipelines in regulated environments. |
| title | A Unified AI System For Data Quality Control and DataOps Management in Regulated Environments |
| topic | Computational Finance |
| url | https://arxiv.org/abs/2512.05559 |