Adaptive Data Flywheel: Applying MAPE Control Loops to AI Agent Improvement
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2025
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866912679153631232 |
|---|---|
| author | Shukla, Aaditya Knowles, Sidney Madugula, Meenakshi Farris, Dave Angilly, Ryan Pombo, Santiago Xu, Anbang An, Lu Balasubramanian, Abhinav Yu, Tan Ren, Jiaxiang Akkiraju, Rama |
| author_facet | Shukla, Aaditya Knowles, Sidney Madugula, Meenakshi Farris, Dave Angilly, Ryan Pombo, Santiago Xu, Anbang An, Lu Balasubramanian, Abhinav Yu, Tan Ren, Jiaxiang Akkiraju, Rama |
| contents | Enterprise AI agents must continuously adapt to maintain accuracy, reduce latency, and remain aligned with user needs. We present a practical implementation of a data flywheel in NVInfo AI, NVIDIA's Mixture-of-Experts (MoE) Knowledge Assistant serving over 30,000 employees. By operationalizing a MAPE-driven data flywheel, we built a closed-loop system that systematically addresses failures in retrieval-augmented generation (RAG) pipelines and enables continuous learning. Over a 3-month post-deployment period, we monitored feedback and collected 495 negative samples. Analysis revealed two major failure modes: routing errors (5.25\%) and query rephrasal errors (3.2\%). Using NVIDIA NeMo microservices, we implemented targeted improvements through fine-tuning. For routing, we replaced a Llama 3.1 70B model with a fine-tuned 8B variant, achieving 96\% accuracy, a 10x reduction in model size, and 70\% latency improvement. For query rephrasal, fine-tuning yielded a 3.7\% gain in accuracy and a 40\% latency reduction. Our approach demonstrates how human-in-the-loop (HITL) feedback, when structured within a data flywheel, transforms enterprise AI agents into self-improving systems. Key learnings include approaches to ensure agent robustness despite limited user feedback, navigating privacy constraints, and executing staged rollouts in production. This work offers a repeatable blueprint for building robust, adaptive enterprise AI agents capable of learning from real-world usage at scale. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_27051 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Adaptive Data Flywheel: Applying MAPE Control Loops to AI Agent Improvement Shukla, Aaditya Knowles, Sidney Madugula, Meenakshi Farris, Dave Angilly, Ryan Pombo, Santiago Xu, Anbang An, Lu Balasubramanian, Abhinav Yu, Tan Ren, Jiaxiang Akkiraju, Rama Artificial Intelligence Machine Learning I.2.6; I.2.11; H.3.3 Enterprise AI agents must continuously adapt to maintain accuracy, reduce latency, and remain aligned with user needs. We present a practical implementation of a data flywheel in NVInfo AI, NVIDIA's Mixture-of-Experts (MoE) Knowledge Assistant serving over 30,000 employees. By operationalizing a MAPE-driven data flywheel, we built a closed-loop system that systematically addresses failures in retrieval-augmented generation (RAG) pipelines and enables continuous learning. Over a 3-month post-deployment period, we monitored feedback and collected 495 negative samples. Analysis revealed two major failure modes: routing errors (5.25\%) and query rephrasal errors (3.2\%). Using NVIDIA NeMo microservices, we implemented targeted improvements through fine-tuning. For routing, we replaced a Llama 3.1 70B model with a fine-tuned 8B variant, achieving 96\% accuracy, a 10x reduction in model size, and 70\% latency improvement. For query rephrasal, fine-tuning yielded a 3.7\% gain in accuracy and a 40\% latency reduction. Our approach demonstrates how human-in-the-loop (HITL) feedback, when structured within a data flywheel, transforms enterprise AI agents into self-improving systems. Key learnings include approaches to ensure agent robustness despite limited user feedback, navigating privacy constraints, and executing staged rollouts in production. This work offers a repeatable blueprint for building robust, adaptive enterprise AI agents capable of learning from real-world usage at scale. |
| title | Adaptive Data Flywheel: Applying MAPE Control Loops to AI Agent Improvement |
| topic | Artificial Intelligence Machine Learning I.2.6; I.2.11; H.3.3 |
| url | https://arxiv.org/abs/2510.27051 |