Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2401.02256 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866914630024036352 |
|---|---|
| author | Tsuta, Yuma Yoshinaga, Naoki Sato, Shoetsu Toyoda, Masashi |
| author_facet | Tsuta, Yuma Yoshinaga, Naoki Sato, Shoetsu Toyoda, Masashi |
| contents | Open-domain dialogue systems have started to engage in continuous conversations with humans. Those dialogue systems are required to be adjusted to the human interlocutor and evaluated in terms of their perspective. However, it is questionable whether the current automatic evaluation methods can approximate the interlocutor's judgments. In this study, we analyzed and examined what features are needed in an automatic response evaluator from the interlocutor's perspective. The first experiment on the Hazumi dataset revealed that interlocutor awareness plays a critical role in making automatic response evaluation correlate with the interlocutor's judgments. The second experiment using massive conversations on X (formerly Twitter) confirmed that dialogue continuity prediction can train an interlocutor-aware response evaluator without human feedback while revealing the difficulty in evaluating generated responses compared to human responses. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2401_02256 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Rethinking Response Evaluation from Interlocutor's Eye for Open-Domain Dialogue Systems Tsuta, Yuma Yoshinaga, Naoki Sato, Shoetsu Toyoda, Masashi Computation and Language Open-domain dialogue systems have started to engage in continuous conversations with humans. Those dialogue systems are required to be adjusted to the human interlocutor and evaluated in terms of their perspective. However, it is questionable whether the current automatic evaluation methods can approximate the interlocutor's judgments. In this study, we analyzed and examined what features are needed in an automatic response evaluator from the interlocutor's perspective. The first experiment on the Hazumi dataset revealed that interlocutor awareness plays a critical role in making automatic response evaluation correlate with the interlocutor's judgments. The second experiment using massive conversations on X (formerly Twitter) confirmed that dialogue continuity prediction can train an interlocutor-aware response evaluator without human feedback while revealing the difficulty in evaluating generated responses compared to human responses. |
| title | Rethinking Response Evaluation from Interlocutor's Eye for Open-Domain Dialogue Systems |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2401.02256 |