Turning Conversations into Workflows: A Framework to Extract and Evaluate Dialog Workflows for Service AI Agents
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2025
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912244023951360 |
|---|---|
| author | Choubey, Prafulla Kumar Peng, Xiangyu Bhagavath, Shilpa Xiong, Caiming Pentyala, Shiva Kumar Wu, Chien-Sheng |
| author_facet | Choubey, Prafulla Kumar Peng, Xiangyu Bhagavath, Shilpa Xiong, Caiming Pentyala, Shiva Kumar Wu, Chien-Sheng |
| contents | Automated service agents require well-structured workflows to provide consistent and accurate responses to customer queries. However, these workflows are often undocumented, and their automatic extraction from conversations remains unexplored. In this work, we present a novel framework for extracting and evaluating dialog workflows from historical interactions. Our extraction process consists of two key stages: (1) a retrieval step to select relevant conversations based on key procedural elements, and (2) a structured workflow generation process using a question-answer-based chain-of-thought (QA-CoT) prompting. To comprehensively assess the quality of extracted workflows, we introduce an automated agent and customer bots simulation framework that measures their effectiveness in resolving customer issues. Extensive experiments on the ABCD and SynthABCD datasets demonstrate that our QA-CoT technique improves workflow extraction by 12.16\% in average macro accuracy over the baseline. Moreover, our evaluation method closely aligns with human assessments, providing a reliable and scalable framework for future research. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2502_17321 |
| institution | arXiv |
| publishDate | 2025 |
| record_format | arxiv |
| spellingShingle | Turning Conversations into Workflows: A Framework to Extract and Evaluate Dialog Workflows for Service AI Agents Choubey, Prafulla Kumar Peng, Xiangyu Bhagavath, Shilpa Xiong, Caiming Pentyala, Shiva Kumar Wu, Chien-Sheng Computation and Language Automated service agents require well-structured workflows to provide consistent and accurate responses to customer queries. However, these workflows are often undocumented, and their automatic extraction from conversations remains unexplored. In this work, we present a novel framework for extracting and evaluating dialog workflows from historical interactions. Our extraction process consists of two key stages: (1) a retrieval step to select relevant conversations based on key procedural elements, and (2) a structured workflow generation process using a question-answer-based chain-of-thought (QA-CoT) prompting. To comprehensively assess the quality of extracted workflows, we introduce an automated agent and customer bots simulation framework that measures their effectiveness in resolving customer issues. Extensive experiments on the ABCD and SynthABCD datasets demonstrate that our QA-CoT technique improves workflow extraction by 12.16\% in average macro accuracy over the baseline. Moreover, our evaluation method closely aligns with human assessments, providing a reliable and scalable framework for future research. |
| title | Turning Conversations into Workflows: A Framework to Extract and Evaluate Dialog Workflows for Service AI Agents |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2502.17321 |