Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | https://arxiv.org/abs/2405.00716 |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866912073980575744 |
|---|---|
| author | Liu, Fenglin Li, Zheng Zhou, Hongjian Yin, Qingyu Yang, Jingfeng Tang, Xianfeng Luo, Chen Zeng, Ming Jiang, Haoming Gao, Yifan Nigam, Priyanka Nag, Sreyashi Yin, Bing Hua, Yining Zhou, Xuan Rohanian, Omid Thakur, Anshul Clifton, Lei Clifton, David A. |
| author_facet | Liu, Fenglin Li, Zheng Zhou, Hongjian Yin, Qingyu Yang, Jingfeng Tang, Xianfeng Luo, Chen Zeng, Ming Jiang, Haoming Gao, Yifan Nigam, Priyanka Nag, Sreyashi Yin, Bing Hua, Yining Zhou, Xuan Rohanian, Omid Thakur, Anshul Clifton, Lei Clifton, David A. |
| contents | The adoption of large language models (LLMs) to assist clinicians has attracted remarkable attention. Existing works mainly adopt the close-ended question-answering (QA) task with answer options for evaluation. However, many clinical decisions involve answering open-ended questions without pre-set options. To better understand LLMs in the clinic, we construct a benchmark ClinicBench. We first collect eleven existing datasets covering diverse clinical language generation, understanding, and reasoning tasks. Furthermore, we construct six novel datasets and clinical tasks that are complex but common in real-world practice, e.g., open-ended decision-making, long document processing, and emerging drug analysis. We conduct an extensive evaluation of twenty-two LLMs under both zero-shot and few-shot settings. Finally, we invite medical experts to evaluate the clinical usefulness of LLMs. The benchmark data is available at https://github.com/AI-in-Health/ClinicBench. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2405_00716 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | Large Language Models in the Clinic: A Comprehensive Benchmark Liu, Fenglin Li, Zheng Zhou, Hongjian Yin, Qingyu Yang, Jingfeng Tang, Xianfeng Luo, Chen Zeng, Ming Jiang, Haoming Gao, Yifan Nigam, Priyanka Nag, Sreyashi Yin, Bing Hua, Yining Zhou, Xuan Rohanian, Omid Thakur, Anshul Clifton, Lei Clifton, David A. Computation and Language Artificial Intelligence The adoption of large language models (LLMs) to assist clinicians has attracted remarkable attention. Existing works mainly adopt the close-ended question-answering (QA) task with answer options for evaluation. However, many clinical decisions involve answering open-ended questions without pre-set options. To better understand LLMs in the clinic, we construct a benchmark ClinicBench. We first collect eleven existing datasets covering diverse clinical language generation, understanding, and reasoning tasks. Furthermore, we construct six novel datasets and clinical tasks that are complex but common in real-world practice, e.g., open-ended decision-making, long document processing, and emerging drug analysis. We conduct an extensive evaluation of twenty-two LLMs under both zero-shot and few-shot settings. Finally, we invite medical experts to evaluate the clinical usefulness of LLMs. The benchmark data is available at https://github.com/AI-in-Health/ClinicBench. |
| title | Large Language Models in the Clinic: A Comprehensive Benchmark |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2405.00716 |