Saved in:
| Main Author: | |
|---|---|
| Format: | Recurso digital |
| Language: | |
| Published: |
Zenodo
2025
|
| Online Access: | https://doi.org/10.5281/zenodo.14872925 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
Table of Contents:
- <p>Generative AI (GenAI) is transforming industries, but its effectiveness depends on access to timely and relevant data. This paper examines the critical role of <strong>real-time data pipelines</strong> in powering GenAI applications, synthesizing existing literature across key areas, including <strong>data integration, streaming platforms, vector databases, and architectural patterns</strong>. We explore the challenges and opportunities in building scalable, high-performance pipelines, emphasizing <strong>data freshness, accuracy, and efficient processing</strong>.</p> <p>The intersection of <strong>GenAI and big data infrastructure</strong> has introduced novel data management techniques, such as <strong>data streaming, integration, and vector databases</strong>, which are crucial for optimizing AI-driven decision-making. This paper reviews these techniques, their applications, and the evolving role of data pipelines in <strong>real-time AI deployment</strong>.</p> <p>A key focus is the integration of <strong>data streaming platforms</strong> with GenAI, enabling real-time processing and enhancing AI applications. We analyze the state-of-the-art in <strong>Apache Kafka, vector databases, and cloud-based solutions</strong>, addressing critical challenges such as <strong>scalability, data consistency, and integration complexity</strong>. Furthermore, we explore future directions, including the use of <strong>retrieval-augmented generation (RAG) and real-time data pipelines</strong> to unlock GenAI’s full potential.</p> <p>This review synthesizes insights from recent research, industry practices, and emerging trends to provide a <strong>comprehensive understanding</strong> of real-time data infrastructure for GenAI.</p>