Saved in:
| Main Authors: | , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2406.17304 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866916299355979776 |
|---|---|
| author | Jia, Jinghan Komma, Abi Leffel, Timothy Peng, Xujun Nagesh, Ajay Soliman, Tamer Galstyan, Aram Kumar, Anoop |
| author_facet | Jia, Jinghan Komma, Abi Leffel, Timothy Peng, Xujun Nagesh, Ajay Soliman, Tamer Galstyan, Aram Kumar, Anoop |
| contents | In task-oriented conversational AI evaluation, unsupervised methods poorly correlate with human judgments, and supervised approaches lack generalization. Recent advances in large language models (LLMs) show robust zeroshot and few-shot capabilities across NLP tasks. This paper explores using LLMs for automated dialogue quality evaluation, experimenting with various configurations on public and proprietary datasets. Manipulating factors such as model size, in-context examples, and selection techniques, we examine "chain-of-thought" (CoT) reasoning and label extraction procedures. Our results show that (1) larger models yield more accurate dialogue labels; (2) algorithmic selection of in-context examples outperforms random selection; (3) CoT reasoning where an LLM is asked to provide justifications before outputting final labels improves performance; and (4) fine-tuned LLMs outperform out-of-the-box ones. Our results indicate that LLMs that are suitably fine-tuned and have sufficient reasoning capabilities can be leveraged for automated dialogue evaluation. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_17304 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Leveraging LLMs for Dialogue Quality Measurement Jia, Jinghan Komma, Abi Leffel, Timothy Peng, Xujun Nagesh, Ajay Soliman, Tamer Galstyan, Aram Kumar, Anoop Computation and Language In task-oriented conversational AI evaluation, unsupervised methods poorly correlate with human judgments, and supervised approaches lack generalization. Recent advances in large language models (LLMs) show robust zeroshot and few-shot capabilities across NLP tasks. This paper explores using LLMs for automated dialogue quality evaluation, experimenting with various configurations on public and proprietary datasets. Manipulating factors such as model size, in-context examples, and selection techniques, we examine "chain-of-thought" (CoT) reasoning and label extraction procedures. Our results show that (1) larger models yield more accurate dialogue labels; (2) algorithmic selection of in-context examples outperforms random selection; (3) CoT reasoning where an LLM is asked to provide justifications before outputting final labels improves performance; and (4) fine-tuned LLMs outperform out-of-the-box ones. Our results indicate that LLMs that are suitably fine-tuned and have sufficient reasoning capabilities can be leveraged for automated dialogue evaluation. |
| title | Leveraging LLMs for Dialogue Quality Measurement |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2406.17304 |