Saved in:
| Main Authors: | , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2408.09831 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912191420039168 |
|---|---|
| author | Heineking, Sebastian Probst, Jonas Steinbach, Daniel Potthast, Martin Scells, Harrisen |
| author_facet | Heineking, Sebastian Probst, Jonas Steinbach, Daniel Potthast, Martin Scells, Harrisen |
| contents | Evaluating the output of generative large language models (LLMs) is challenging and difficult to scale. Many evaluations of LLMs focus on tasks such as single-choice question-answering or text classification. These tasks are not suitable for assessing open-ended question-answering capabilities, which are critical in domains where expertise is required. One such domain is health, where misleading or incorrect answers can have a negative impact on a user's well-being. Using human experts to evaluate the quality of LLM answers is generally considered the gold standard, but expert annotation is costly and slow. We present a method for evaluating LLM answers that uses ranking models trained on annotated document collections as a substitute for explicit relevance judgements and apply it to the CLEF 2021 eHealth dataset. In a user study, our method correlates with the preferences of a human expert (Kendall's $τ=0.64$). It is also consistent with previous findings in that the quality of generated answers improves with the size of the model and more sophisticated prompting strategies. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2408_09831 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Ranking Generated Answers: On the Agreement of Retrieval Models with Humans on Consumer Health Questions Heineking, Sebastian Probst, Jonas Steinbach, Daniel Potthast, Martin Scells, Harrisen Information Retrieval Evaluating the output of generative large language models (LLMs) is challenging and difficult to scale. Many evaluations of LLMs focus on tasks such as single-choice question-answering or text classification. These tasks are not suitable for assessing open-ended question-answering capabilities, which are critical in domains where expertise is required. One such domain is health, where misleading or incorrect answers can have a negative impact on a user's well-being. Using human experts to evaluate the quality of LLM answers is generally considered the gold standard, but expert annotation is costly and slow. We present a method for evaluating LLM answers that uses ranking models trained on annotated document collections as a substitute for explicit relevance judgements and apply it to the CLEF 2021 eHealth dataset. In a user study, our method correlates with the preferences of a human expert (Kendall's $τ=0.64$). It is also consistent with previous findings in that the quality of generated answers improves with the size of the model and more sophisticated prompting strategies. |
| title | Ranking Generated Answers: On the Agreement of Retrieval Models with Humans on Consumer Health Questions |
| topic | Information Retrieval |
| url | https://arxiv.org/abs/2408.09831 |