Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2025
|
| Schlagworte: | |
| Online-Zugang: | https://arxiv.org/abs/2510.06674 |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866915540753186816 |
|---|---|
| author | Zhao, Cen Mia Zhang, Tiantian Su, Hanchen Zhang, Yufeng Wayne Su, Shaowei Xu, Mingzhi Liu, Yu Elaine Han, Wei Werner, Jeremy Cheng, Claire Na Mehdad, Yashar |
| author_facet | Zhao, Cen Mia Zhang, Tiantian Su, Hanchen Zhang, Yufeng Wayne Su, Shaowei Xu, Mingzhi Liu, Yu Elaine Han, Wei Werner, Jeremy Cheng, Claire Na Mehdad, Yashar |
| contents | We introduce an Agent-in-the-Loop (AITL) framework that implements a continuous data flywheel for iteratively improving an LLM-based customer support system. Unlike standard offline approaches that rely on batch annotations, AITL integrates four key types of annotations directly into live customer operations: (1) pairwise response preferences, (2) agent adoption and rationales, (3) knowledge relevance checks, and (4) identification of missing knowledge. These feedback signals seamlessly feed back into models' updates, reducing retraining cycles from months to weeks. Our production pilot involving US-based customer support agents demonstrated significant improvements in retrieval accuracy (+11.7% recall@75, +14.8% precision@8), generation quality (+8.4% helpfulness) and agent adoption rates (+4.5%). These results underscore the effectiveness of embedding human feedback loops directly into operational workflows to continuously refine LLM-based customer support system. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2510_06674 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Agent-in-the-Loop: A Data Flywheel for Continuous Improvement in LLM-based Customer Support Zhao, Cen Mia Zhang, Tiantian Su, Hanchen Zhang, Yufeng Wayne Su, Shaowei Xu, Mingzhi Liu, Yu Elaine Han, Wei Werner, Jeremy Cheng, Claire Na Mehdad, Yashar Artificial Intelligence We introduce an Agent-in-the-Loop (AITL) framework that implements a continuous data flywheel for iteratively improving an LLM-based customer support system. Unlike standard offline approaches that rely on batch annotations, AITL integrates four key types of annotations directly into live customer operations: (1) pairwise response preferences, (2) agent adoption and rationales, (3) knowledge relevance checks, and (4) identification of missing knowledge. These feedback signals seamlessly feed back into models' updates, reducing retraining cycles from months to weeks. Our production pilot involving US-based customer support agents demonstrated significant improvements in retrieval accuracy (+11.7% recall@75, +14.8% precision@8), generation quality (+8.4% helpfulness) and agent adoption rates (+4.5%). These results underscore the effectiveness of embedding human feedback loops directly into operational workflows to continuously refine LLM-based customer support system. |
| title | Agent-in-the-Loop: A Data Flywheel for Continuous Improvement in LLM-based Customer Support |
| topic | Artificial Intelligence |
| url | https://arxiv.org/abs/2510.06674 |