Reliable LLM-based User Simulator for Task-Oriented Dialogue Systems
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866917594818150400 |
|---|---|
| author | Sekulić, Ivan Terragni, Silvia Guimarães, Victor Khau, Nghia Guedes, Bruna Filipavicius, Modestas Manso, André Ferreira Mathis, Roland |
| author_facet | Sekulić, Ivan Terragni, Silvia Guimarães, Victor Khau, Nghia Guedes, Bruna Filipavicius, Modestas Manso, André Ferreira Mathis, Roland |
| contents | In the realm of dialogue systems, user simulation techniques have emerged as a game-changer, redefining the evaluation and enhancement of task-oriented dialogue (TOD) systems. These methods are crucial for replicating real user interactions, enabling applications like synthetic data augmentation, error detection, and robust evaluation. However, existing approaches often rely on rigid rule-based methods or on annotated data. This paper introduces DAUS, a Domain-Aware User Simulator. Leveraging large language models, we fine-tune DAUS on real examples of task-oriented dialogues. Results on two relevant benchmarks showcase significant improvements in terms of user goal fulfillment. Notably, we have observed that fine-tuning enhances the simulator's coherence with user goals, effectively mitigating hallucinations -- a major source of inconsistencies in simulator responses. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2402_13374 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Reliable LLM-based User Simulator for Task-Oriented Dialogue Systems Sekulić, Ivan Terragni, Silvia Guimarães, Victor Khau, Nghia Guedes, Bruna Filipavicius, Modestas Manso, André Ferreira Mathis, Roland Computation and Language In the realm of dialogue systems, user simulation techniques have emerged as a game-changer, redefining the evaluation and enhancement of task-oriented dialogue (TOD) systems. These methods are crucial for replicating real user interactions, enabling applications like synthetic data augmentation, error detection, and robust evaluation. However, existing approaches often rely on rigid rule-based methods or on annotated data. This paper introduces DAUS, a Domain-Aware User Simulator. Leveraging large language models, we fine-tune DAUS on real examples of task-oriented dialogues. Results on two relevant benchmarks showcase significant improvements in terms of user goal fulfillment. Notably, we have observed that fine-tuning enhances the simulator's coherence with user goals, effectively mitigating hallucinations -- a major source of inconsistencies in simulator responses. |
| title | Reliable LLM-based User Simulator for Task-Oriented Dialogue Systems |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2402.13374 |