LLM as a Scorer: The Impact of Output Order on Dialogue Evaluation
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866911905084342272 |
|---|---|
| author | Chen, Yi-Pei Chu, KuanChao Nakayama, Hideki |
| author_facet | Chen, Yi-Pei Chu, KuanChao Nakayama, Hideki |
| contents | This research investigates the effect of prompt design on dialogue evaluation using large language models (LLMs). While LLMs are increasingly used for scoring various inputs, creating effective prompts for dialogue evaluation remains challenging due to model sensitivity and subjectivity in dialogue assessments. Our study experimented with different prompt structures, altering the sequence of output instructions and including explanatory reasons. We found that the order of presenting reasons and scores significantly influences LLMs' scoring, with a "reason-first" approach yielding more comprehensive evaluations. This insight is crucial for enhancing the accuracy and consistency of LLM-based evaluations. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_02863 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | LLM as a Scorer: The Impact of Output Order on Dialogue Evaluation Chen, Yi-Pei Chu, KuanChao Nakayama, Hideki Computation and Language This research investigates the effect of prompt design on dialogue evaluation using large language models (LLMs). While LLMs are increasingly used for scoring various inputs, creating effective prompts for dialogue evaluation remains challenging due to model sensitivity and subjectivity in dialogue assessments. Our study experimented with different prompt structures, altering the sequence of output instructions and including explanatory reasons. We found that the order of presenting reasons and scores significantly influences LLMs' scoring, with a "reason-first" approach yielding more comprehensive evaluations. This insight is crucial for enhancing the accuracy and consistency of LLM-based evaluations. |
| title | LLM as a Scorer: The Impact of Output Order on Dialogue Evaluation |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2406.02863 |