Saved in:
| Main Authors: | , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2404.01015 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910532907302912 |
|---|---|
| author | Park, ChaeHun Choi, Minseok Lee, Dohyun Choo, Jaegul |
| author_facet | Park, ChaeHun Choi, Minseok Lee, Dohyun Choo, Jaegul |
| contents | Building a reliable and automated evaluation metric is a necessary but challenging problem for open-domain dialogue systems. Recent studies proposed evaluation metrics that assess generated responses by considering their relevance to previous dialogue histories. Although effective, these metrics evaluate individual responses directly rather than considering their relative quality compared to other responses. To handle this, we propose PairEval, a novel dialogue evaluation metric for assessing responses by comparing their quality against responses in different conversations. PairEval is built on top of open-sourced and moderate-size language models, and we make them specialized in pairwise comparison between dialogue responses. Extensive experiments on multiple benchmarks demonstrate that our metric exhibits a higher correlation with human judgments than baseline metrics. We also find that the proposed comparative metric is more robust in detecting common failures from open-domain dialogue systems, including repetition and speaker insensitivity. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2404_01015 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | PairEval: Open-domain Dialogue Evaluation with Pairwise Comparison Park, ChaeHun Choi, Minseok Lee, Dohyun Choo, Jaegul Computation and Language Building a reliable and automated evaluation metric is a necessary but challenging problem for open-domain dialogue systems. Recent studies proposed evaluation metrics that assess generated responses by considering their relevance to previous dialogue histories. Although effective, these metrics evaluate individual responses directly rather than considering their relative quality compared to other responses. To handle this, we propose PairEval, a novel dialogue evaluation metric for assessing responses by comparing their quality against responses in different conversations. PairEval is built on top of open-sourced and moderate-size language models, and we make them specialized in pairwise comparison between dialogue responses. Extensive experiments on multiple benchmarks demonstrate that our metric exhibits a higher correlation with human judgments than baseline metrics. We also find that the proposed comparative metric is more robust in detecting common failures from open-domain dialogue systems, including repetition and speaker insensitivity. |
| title | PairEval: Open-domain Dialogue Evaluation with Pairwise Comparison |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2404.01015 |