MedCalc-Bench: Evaluating Large Language Models for Medical Calculations
Fuente:
arXiv
Saved in:
| Main Authors: | , , , , , , , , , , , , , , , , |
|---|---|
| Format: | Preprint |
| Published: |
2024
|
| Subjects: | |
| Online Access: | |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| _version_ | 1866910507389157376 |
|---|---|
| author | Khandekar, Nikhil Jin, Qiao Xiong, Guangzhi Dunn, Soren Applebaum, Serina S Anwar, Zain Sarfo-Gyamfi, Maame Safranek, Conrad W Anwar, Abid A Zhang, Andrew Gilson, Aidan Singer, Maxwell B Dave, Amisha Taylor, Andrew Zhang, Aidong Chen, Qingyu Lu, Zhiyong |
| author_facet | Khandekar, Nikhil Jin, Qiao Xiong, Guangzhi Dunn, Soren Applebaum, Serina S Anwar, Zain Sarfo-Gyamfi, Maame Safranek, Conrad W Anwar, Abid A Zhang, Andrew Gilson, Aidan Singer, Maxwell B Dave, Amisha Taylor, Andrew Zhang, Aidong Chen, Qingyu Lu, Zhiyong |
| contents | As opposed to evaluating computation and logic-based reasoning, current benchmarks for evaluating large language models (LLMs) in medicine are primarily focused on question-answering involving domain knowledge and descriptive reasoning. While such qualitative capabilities are vital to medical diagnosis, in real-world scenarios, doctors frequently use clinical calculators that follow quantitative equations and rule-based reasoning paradigms for evidence-based decision support. To this end, we propose MedCalc-Bench, a first-of-its-kind dataset focused on evaluating the medical calculation capability of LLMs. MedCalc-Bench contains an evaluation set of over 1000 manually reviewed instances from 55 different medical calculation tasks. Each instance in MedCalc-Bench consists of a patient note, a question requesting to compute a specific medical value, a ground truth answer, and a step-by-step explanation showing how the answer is obtained. While our evaluation results show the potential of LLMs in this area, none of them are effective enough for clinical settings. Common issues include extracting the incorrect entities, not using the correct equation or rules for a calculation task, or incorrectly performing the arithmetic for the computation. We hope our study highlights the quantitative knowledge and reasoning gaps in LLMs within medical settings, encouraging future improvements of LLMs for various clinical calculation tasks. |
| format | Preprint |
| id |
arxiv_https___arxiv_org_abs_2406_12036 |
| institution | arXiv |
| publishDate | 2024 |
| record_format | arxiv |
| spellingShingle | MedCalc-Bench: Evaluating Large Language Models for Medical Calculations Khandekar, Nikhil Jin, Qiao Xiong, Guangzhi Dunn, Soren Applebaum, Serina S Anwar, Zain Sarfo-Gyamfi, Maame Safranek, Conrad W Anwar, Abid A Zhang, Andrew Gilson, Aidan Singer, Maxwell B Dave, Amisha Taylor, Andrew Zhang, Aidong Chen, Qingyu Lu, Zhiyong Computation and Language Artificial Intelligence As opposed to evaluating computation and logic-based reasoning, current benchmarks for evaluating large language models (LLMs) in medicine are primarily focused on question-answering involving domain knowledge and descriptive reasoning. While such qualitative capabilities are vital to medical diagnosis, in real-world scenarios, doctors frequently use clinical calculators that follow quantitative equations and rule-based reasoning paradigms for evidence-based decision support. To this end, we propose MedCalc-Bench, a first-of-its-kind dataset focused on evaluating the medical calculation capability of LLMs. MedCalc-Bench contains an evaluation set of over 1000 manually reviewed instances from 55 different medical calculation tasks. Each instance in MedCalc-Bench consists of a patient note, a question requesting to compute a specific medical value, a ground truth answer, and a step-by-step explanation showing how the answer is obtained. While our evaluation results show the potential of LLMs in this area, none of them are effective enough for clinical settings. Common issues include extracting the incorrect entities, not using the correct equation or rules for a calculation task, or incorrectly performing the arithmetic for the computation. We hope our study highlights the quantitative knowledge and reasoning gaps in LLMs within medical settings, encouraging future improvements of LLMs for various clinical calculation tasks. |
| title | MedCalc-Bench: Evaluating Large Language Models for Medical Calculations |
| topic | Computation and Language Artificial Intelligence |
| url | https://arxiv.org/abs/2406.12036 |