Interpretable User Satisfaction Estimation for Conversational Systems with Large Language Models
Fuente:
arXiv
Gespeichert in:
| Hauptverfasser: | , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Veröffentlicht: |
2024
|
| Schlagworte: | |
| Online-Zugang: | |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| _version_ | 1866911910437322752 |
|---|---|
| author | Lin, Ying-Chun Neville, Jennifer Stokes, Jack W. Yang, Longqi Safavi, Tara Wan, Mengting Counts, Scott Suri, Siddharth Andersen, Reid Xu, Xiaofeng Gupta, Deepak Jauhar, Sujay Kumar Song, Xia Buscher, Georg Tiwary, Saurabh Hecht, Brent Teevan, Jaime |
| author_facet | Lin, Ying-Chun Neville, Jennifer Stokes, Jack W. Yang, Longqi Safavi, Tara Wan, Mengting Counts, Scott Suri, Siddharth Andersen, Reid Xu, Xiaofeng Gupta, Deepak Jauhar, Sujay Kumar Song, Xia Buscher, Georg Tiwary, Saurabh Hecht, Brent Teevan, Jaime |
| contents | Accurate and interpretable user satisfaction estimation (USE) is critical for understanding, evaluating, and continuously improving conversational systems. Users express their satisfaction or dissatisfaction with diverse conversational patterns in both general-purpose (ChatGPT and Bing Copilot) and task-oriented (customer service chatbot) conversational systems. Existing approaches based on featurized ML models or text embeddings fall short in extracting generalizable patterns and are hard to interpret. In this work, we show that LLMs can extract interpretable signals of user satisfaction from their natural language utterances more effectively than embedding-based approaches. Moreover, an LLM can be tailored for USE via an iterative prompting framework using supervision from labeled examples. The resulting method, Supervised Prompting for User satisfaction Rubrics (SPUR), not only has higher accuracy but is more interpretable as it scores user satisfaction via learned rubrics with a detailed breakdown. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_12388 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Interpretable User Satisfaction Estimation for Conversational Systems with Large Language Models Lin, Ying-Chun Neville, Jennifer Stokes, Jack W. Yang, Longqi Safavi, Tara Wan, Mengting Counts, Scott Suri, Siddharth Andersen, Reid Xu, Xiaofeng Gupta, Deepak Jauhar, Sujay Kumar Song, Xia Buscher, Georg Tiwary, Saurabh Hecht, Brent Teevan, Jaime Information Retrieval Artificial Intelligence Accurate and interpretable user satisfaction estimation (USE) is critical for understanding, evaluating, and continuously improving conversational systems. Users express their satisfaction or dissatisfaction with diverse conversational patterns in both general-purpose (ChatGPT and Bing Copilot) and task-oriented (customer service chatbot) conversational systems. Existing approaches based on featurized ML models or text embeddings fall short in extracting generalizable patterns and are hard to interpret. In this work, we show that LLMs can extract interpretable signals of user satisfaction from their natural language utterances more effectively than embedding-based approaches. Moreover, an LLM can be tailored for USE via an iterative prompting framework using supervision from labeled examples. The resulting method, Supervised Prompting for User satisfaction Rubrics (SPUR), not only has higher accuracy but is more interpretable as it scores user satisfaction via learned rubrics with a detailed breakdown. |
| title | Interpretable User Satisfaction Estimation for Conversational Systems with Large Language Models |
| topic | Information Retrieval Artificial Intelligence |
| url | https://arxiv.org/abs/2403.12388 |