Saved in:
| Main Authors: | , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2406.02863 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866911905084342272 |
|---|---|
| author | Chen, Yi-Pei Chu, KuanChao Nakayama, Hideki |
| author_facet | Chen, Yi-Pei Chu, KuanChao Nakayama, Hideki |
| contents | This research investigates the effect of prompt design on dialogue evaluation using large language models (LLMs). While LLMs are increasingly used for scoring various inputs, creating effective prompts for dialogue evaluation remains challenging due to model sensitivity and subjectivity in dialogue assessments. Our study experimented with different prompt structures, altering the sequence of output instructions and including explanatory reasons. We found that the order of presenting reasons and scores significantly influences LLMs' scoring, with a "reason-first" approach yielding more comprehensive evaluations. This insight is crucial for enhancing the accuracy and consistency of LLM-based evaluations. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_02863 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | LLM as a Scorer: The Impact of Output Order on Dialogue Evaluation Chen, Yi-Pei Chu, KuanChao Nakayama, Hideki Computation and Language This research investigates the effect of prompt design on dialogue evaluation using large language models (LLMs). While LLMs are increasingly used for scoring various inputs, creating effective prompts for dialogue evaluation remains challenging due to model sensitivity and subjectivity in dialogue assessments. Our study experimented with different prompt structures, altering the sequence of output instructions and including explanatory reasons. We found that the order of presenting reasons and scores significantly influences LLMs' scoring, with a "reason-first" approach yielding more comprehensive evaluations. This insight is crucial for enhancing the accuracy and consistency of LLM-based evaluations. |
| title | LLM as a Scorer: The Impact of Output Order on Dialogue Evaluation |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2406.02863 |