A Better LLM Evaluator for Text Generation: The Impact of Prompt Output Sequencing and Optimization
Fuente:
arXiv
Salvato in:
| Autori principali: | , , |
|---|---|
| Natura: | Preprint |
| Pubblicazione: |
2024
|
| Soggetti: | |
| Accesso online: | |
| Tags: |
Aggiungi Tag
Nessun Tag, puoi essere il primo ad aggiungerne!!
|
| _version_ | 1866911934505287680 |
|---|---|
| author | Chu, KuanChao Chen, Yi-Pei Nakayama, Hideki |
| author_facet | Chu, KuanChao Chen, Yi-Pei Nakayama, Hideki |
| contents | This research investigates prompt designs of evaluating generated texts using large language models (LLMs). While LLMs are increasingly used for scoring various inputs, creating effective prompts for open-ended text evaluation remains challenging due to model sensitivity and subjectivity in evaluation of text generation. Our study experimented with different prompt structures, altering the sequence of output instructions and including explanatory reasons. We found that the order of presenting reasons and scores significantly influences LLMs' scoring, with a different level of rule understanding in the prompt. An additional optimization may enhance scoring alignment if sufficient data is available. This insight is crucial for improving the accuracy and consistency of LLM-based evaluations. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_09972 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | A Better LLM Evaluator for Text Generation: The Impact of Prompt Output Sequencing and Optimization Chu, KuanChao Chen, Yi-Pei Nakayama, Hideki Computation and Language This research investigates prompt designs of evaluating generated texts using large language models (LLMs). While LLMs are increasingly used for scoring various inputs, creating effective prompts for open-ended text evaluation remains challenging due to model sensitivity and subjectivity in evaluation of text generation. Our study experimented with different prompt structures, altering the sequence of output instructions and including explanatory reasons. We found that the order of presenting reasons and scores significantly influences LLMs' scoring, with a different level of rule understanding in the prompt. An additional optimization may enhance scoring alignment if sufficient data is available. This insight is crucial for improving the accuracy and consistency of LLM-based evaluations. |
| title | A Better LLM Evaluator for Text Generation: The Impact of Prompt Output Sequencing and Optimization |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2406.09972 |