"Stupid robot, I want to speak to a human!" User Frustration Detection in Task-Oriented Dialog Systems
Fuente:
arXiv
Salvato in:
| Autori principali: | , , , , , , , , , , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866929669597560832 |
|---|---|
| author | Caralt, Mireia Hernandez Sekulić, Ivan Carević, Filip Khau, Nghia Popa, Diana Nicoleta Guedes, Bruna Guimarães, Victor Yang, Zeyu Manso, Andre Reddy, Meghana Rosso, Paolo Mathis, Roland |
| author_facet | Caralt, Mireia Hernandez Sekulić, Ivan Carević, Filip Khau, Nghia Popa, Diana Nicoleta Guedes, Bruna Guimarães, Victor Yang, Zeyu Manso, Andre Reddy, Meghana Rosso, Paolo Mathis, Roland |
| contents | Detecting user frustration in modern-day task-oriented dialog (TOD) systems is imperative for maintaining overall user satisfaction, engagement, and retention. However, most recent research is focused on sentiment and emotion detection in academic settings, thus failing to fully encapsulate implications of real-world user data. To mitigate this gap, in this work, we focus on user frustration in a deployed TOD system, assessing the feasibility of out-of-the-box solutions for user frustration detection. Specifically, we compare the performance of our deployed keyword-based approach, open-source approaches to sentiment analysis, dialog breakdown detection methods, and emerging in-context learning LLM-based detection. Our analysis highlights the limitations of open-source methods for real-world frustration detection, while demonstrating the superior performance of the LLM-based approach, achieving a 16\% relative improvement in F1 score on an internal benchmark. Finally, we analyze advantages and limitations of our methods and provide an insight into user frustration detection task for industry practitioners. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2411_17437 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | "Stupid robot, I want to speak to a human!" User Frustration Detection in Task-Oriented Dialog Systems Caralt, Mireia Hernandez Sekulić, Ivan Carević, Filip Khau, Nghia Popa, Diana Nicoleta Guedes, Bruna Guimarães, Victor Yang, Zeyu Manso, Andre Reddy, Meghana Rosso, Paolo Mathis, Roland Computation and Language Detecting user frustration in modern-day task-oriented dialog (TOD) systems is imperative for maintaining overall user satisfaction, engagement, and retention. However, most recent research is focused on sentiment and emotion detection in academic settings, thus failing to fully encapsulate implications of real-world user data. To mitigate this gap, in this work, we focus on user frustration in a deployed TOD system, assessing the feasibility of out-of-the-box solutions for user frustration detection. Specifically, we compare the performance of our deployed keyword-based approach, open-source approaches to sentiment analysis, dialog breakdown detection methods, and emerging in-context learning LLM-based detection. Our analysis highlights the limitations of open-source methods for real-world frustration detection, while demonstrating the superior performance of the LLM-based approach, achieving a 16\% relative improvement in F1 score on an internal benchmark. Finally, we analyze advantages and limitations of our methods and provide an insight into user frustration detection task for industry practitioners. |
| title | "Stupid robot, I want to speak to a human!" User Frustration Detection in Task-Oriented Dialog Systems |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2411.17437 |