A Framework for Human Evaluation of Large Language Models in Healthcare Derived from Literature Review
Fuente:
arXiv
Guardado en:
| Autores principales: | , , , , , , , , , , , , , , |
|---|---|
| Formato: | Preprint |
| Publicado: |
2024
|
| Materias: | |
| Acceso en línea: | |
| Etiquetas: |
Agregar Etiqueta
Sin Etiquetas, Sea el primero en etiquetar este registro!
|
| _version_ | 1866909323712528384 |
|---|---|
| author | Tam, Thomas Yu Chow Sivarajkumar, Sonish Kapoor, Sumit Stolyar, Alisa V Polanska, Katelyn McCarthy, Karleigh R Osterhoudt, Hunter Wu, Xizhi Visweswaran, Shyam Fu, Sunyang Mathur, Piyush Cacciamani, Giovanni E. Sun, Cong Peng, Yifan Wang, Yanshan |
| author_facet | Tam, Thomas Yu Chow Sivarajkumar, Sonish Kapoor, Sumit Stolyar, Alisa V Polanska, Katelyn McCarthy, Karleigh R Osterhoudt, Hunter Wu, Xizhi Visweswaran, Shyam Fu, Sunyang Mathur, Piyush Cacciamani, Giovanni E. Sun, Cong Peng, Yifan Wang, Yanshan |
| contents | With generative artificial intelligence (AI), particularly large language models (LLMs), continuing to make inroads in healthcare, it is critical to supplement traditional automated evaluations with human evaluations. Understanding and evaluating the output of LLMs is essential to assuring safety, reliability, and effectiveness. However, human evaluation's cumbersome, time-consuming, and non-standardized nature presents significant obstacles to comprehensive evaluation and widespread adoption of LLMs in practice. This study reviews existing literature on human evaluation methodologies for LLMs in healthcare. We highlight a notable need for a standardized and consistent human evaluation approach. Our extensive literature search, adhering to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines, includes publications from January 2018 to February 2024. The review examines the human evaluation of LLMs across various medical specialties, addressing factors such as evaluation dimensions, sample types and sizes, selection, and recruitment of evaluators, frameworks and metrics, evaluation process, and statistical analysis type. Drawing on the diverse evaluation strategies employed in these studies, we propose a comprehensive and practical framework for human evaluation of LLMs: QUEST: Quality of Information, Understanding and Reasoning, Expression Style and Persona, Safety and Harm, and Trust and Confidence. This framework aims to improve the reliability, generalizability, and applicability of human evaluation of LLMs in different healthcare applications by defining clear evaluation dimensions and offering detailed guidelines. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_02559 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | A Framework for Human Evaluation of Large Language Models in Healthcare Derived from Literature Review Tam, Thomas Yu Chow Sivarajkumar, Sonish Kapoor, Sumit Stolyar, Alisa V Polanska, Katelyn McCarthy, Karleigh R Osterhoudt, Hunter Wu, Xizhi Visweswaran, Shyam Fu, Sunyang Mathur, Piyush Cacciamani, Giovanni E. Sun, Cong Peng, Yifan Wang, Yanshan Computation and Language Artificial Intelligence With generative artificial intelligence (AI), particularly large language models (LLMs), continuing to make inroads in healthcare, it is critical to supplement traditional automated evaluations with human evaluations. Understanding and evaluating the output of LLMs is essential to assuring safety, reliability, and effectiveness. However, human evaluation's cumbersome, time-consuming, and non-standardized nature presents significant obstacles to comprehensive evaluation and widespread adoption of LLMs in practice. This study reviews existing literature on human evaluation methodologies for LLMs in healthcare. We highlight a notable need for a standardized and consistent human evaluation approach. Our extensive literature search, adhering to the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines, includes publications from January 2018 to February 2024. The review examines the human evaluation of LLMs across various medical specialties, addressing factors such as evaluation dimensions, sample types and sizes, selection, and recruitment of evaluators, frameworks and metrics, evaluation process, and statistical analysis type. Drawing on the diverse evaluation strategies employed in these studies, we propose a comprehensive and practical framework for human evaluation of LLMs: QUEST: Quality of Information, Understanding and Reasoning, Expression Style and Persona, Safety and Harm, and Trust and Confidence. This framework aims to improve the reliability, generalizability, and applicability of human evaluation of LLMs in different healthcare applications by defining clear evaluation dimensions and offering detailed guidelines. |
| title | A Framework for Human Evaluation of Large Language Models in Healthcare Derived from Literature Review |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2405.02559 |