CheckEval: A reliable LLM-as-a-Judge framework for evaluating text generation using checklists
Fuente:
arXiv
Enregistré dans:
| Auteurs principaux: | , , , , , , |
|---|---|
| Format: | Preprint |
| Publié: |
2024
|
| Sujets: | |
| Accès en ligne: | |
| Tags: |
Ajouter un tag
Pas de tags, Soyez le premier à ajouter un tag!
|
| _version_ | 1866909880645844992 |
|---|---|
| author | Lee, Yukyung Kim, Joonghoon Kim, Jaehee Cho, Hyowon Kang, Jaewook Kang, Pilsung Kim, Najoung |
| author_facet | Lee, Yukyung Kim, Joonghoon Kim, Jaehee Cho, Hyowon Kang, Jaewook Kang, Pilsung Kim, Najoung |
| contents | Existing LLM-as-a-Judge approaches for evaluating text generation suffer from rating inconsistencies, with low agreement and high rating variance across different evaluator models. We attribute this to subjective evaluation criteria combined with Likert scale scoring in existing protocols. To address this issue, we introduce CheckEval, a checklist-based evaluation framework that improves rating reliability via decomposed binary questions. Through experiments with 12 evaluator models across multiple datasets, we first demonstrate that CheckEval strongly correlates with human judgments. More importantly, CheckEval dramatically improves the average agreement across evaluator models by 0.45 and reduces the score variance. CheckEval scores furthermore have the benefit of being more interpretable because it decomposes evaluation criteria into traceable binary decisions, allowing analyses of specific attributes driving quality judgments. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2403_18771 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | CheckEval: A reliable LLM-as-a-Judge framework for evaluating text generation using checklists Lee, Yukyung Kim, Joonghoon Kim, Jaehee Cho, Hyowon Kang, Jaewook Kang, Pilsung Kim, Najoung Computation and Language Existing LLM-as-a-Judge approaches for evaluating text generation suffer from rating inconsistencies, with low agreement and high rating variance across different evaluator models. We attribute this to subjective evaluation criteria combined with Likert scale scoring in existing protocols. To address this issue, we introduce CheckEval, a checklist-based evaluation framework that improves rating reliability via decomposed binary questions. Through experiments with 12 evaluator models across multiple datasets, we first demonstrate that CheckEval strongly correlates with human judgments. More importantly, CheckEval dramatically improves the average agreement across evaluator models by 0.45 and reduces the score variance. CheckEval scores furthermore have the benefit of being more interpretable because it decomposes evaluation criteria into traceable binary decisions, allowing analyses of specific attributes driving quality judgments. |
| title | CheckEval: A reliable LLM-as-a-Judge framework for evaluating text generation using checklists |
| topic | Computation and Language |
| url | https://arxiv.org/abs/2403.18771 |